[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:41 dist-test-slave-8tg2 krb5kdc[30340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:41 dist-test-slave-8tg2 krb5kdc[30340](info): set up 2 sockets
Sep 27 18:46:41 dist-test-slave-8tg2 krb5kdc[30340](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 18:46:42.209102 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:46:42 dist-test-slave-8tg2 krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:46:42.334800 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44083
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:44083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:42.446595 30353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:42.446866 30353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:42.446906 30353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:42.451036 30353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:42.451097 30353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:42.451116 30353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:42.451133 30353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:42.451171 30353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:42.456429 30353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:44083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:42.457453 30353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:42.458590 30353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:46:42.467741 30353 server_base.cc:1034] running on GCE node
W20260927 18:46:42.467872 30358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:42.467813 30361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:42.468093 30359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:42.468673 30353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:42.469748 30353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:42.470902 30353 hybrid_clock.cc:648] HybridClock initialized: now 1790534802470872 us; error 42 us; skew 500 ppm
Sep 27 18:46:42 dist-test-slave-8tg2 krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534802, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:42.473573 30353 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:46:42.474439 30353 webserver.cc:460] Webserver started at http://127.29.159.126:43091/ using document root <none> and password file <none>
I20260927 18:46:42.474975 30353 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:42.475035 30353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:42.475271 30353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:42.477324 30353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "95661d1e5e1d49539c42674b71d21421"
format_stamp: "Formatted at 2026-09-27 18:46:42 on dist-test-slave-8tg2"
server_key: "f5a9b671c2795680ead9442df35eb1a3"
server_key_iv: "0b0c28fb97a76f0fb0d3d37c0e116a38"
server_key_version: "encryptionkey@0"
I20260927 18:46:42.477890 30353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "95661d1e5e1d49539c42674b71d21421"
format_stamp: "Formatted at 2026-09-27 18:46:42 on dist-test-slave-8tg2"
server_key: "f5a9b671c2795680ead9442df35eb1a3"
server_key_iv: "0b0c28fb97a76f0fb0d3d37c0e116a38"
server_key_version: "encryptionkey@0"
I20260927 18:46:42.481698 30353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:46:42.484045 30368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:42.485232 30353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:42.485350 30353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "95661d1e5e1d49539c42674b71d21421"
format_stamp: "Formatted at 2026-09-27 18:46:42 on dist-test-slave-8tg2"
server_key: "f5a9b671c2795680ead9442df35eb1a3"
server_key_iv: "0b0c28fb97a76f0fb0d3d37c0e116a38"
server_key_version: "encryptionkey@0"
I20260927 18:46:42.485446 30353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:42.501745 30353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:42.504717 30353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:42.504916 30353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:42.511925 30353 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:44083
I20260927 18:46:42.511950 30420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:44083 every 8 connection(s)
I20260927 18:46:42.512831 30353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:42.515581 30421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:42.520879 30421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421: Bootstrap starting.
I20260927 18:46:42.521589 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 30353
I20260927 18:46:42.521771 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:46:42.522984 30333 external_mini_cluster.cc:1428] Setting key df839c5be8537caac0f36e07d9749b89
I20260927 18:46:42.523308 30421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:42.524022 30421 log.cc:824] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:42.525978 30421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421: No bootstrap required, opened a new log
I20260927 18:46:42.531014 30421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95661d1e5e1d49539c42674b71d21421" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44083 } }
I20260927 18:46:42.531289 30421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:42.531401 30421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95661d1e5e1d49539c42674b71d21421, State: Initialized, Role: FOLLOWER
I20260927 18:46:42.531869 30421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95661d1e5e1d49539c42674b71d21421" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44083 } }
I20260927 18:46:42.531984 30421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:42.532068 30421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:42.532189 30421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:42.533352 30421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95661d1e5e1d49539c42674b71d21421" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44083 } }
I20260927 18:46:42.533713 30421 leader_election.cc:302] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95661d1e5e1d49539c42674b71d21421; no voters:
I20260927 18:46:42.534058 30421 leader_election.cc:288] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:42.534147 30426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:42.534379 30426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [term 1 LEADER]: Becoming Leader. State: Replica: 95661d1e5e1d49539c42674b71d21421, State: Running, Role: LEADER
I20260927 18:46:42.534718 30426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95661d1e5e1d49539c42674b71d21421" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44083 } }
I20260927 18:46:42.535329 30421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:42.536382 30427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95661d1e5e1d49539c42674b71d21421" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95661d1e5e1d49539c42674b71d21421" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44083 } } }
I20260927 18:46:42.536504 30427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:42.536859 30428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95661d1e5e1d49539c42674b71d21421. Latest consensus state: current_term: 1 leader_uuid: "95661d1e5e1d49539c42674b71d21421" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95661d1e5e1d49539c42674b71d21421" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44083 } } }
I20260927 18:46:42.537004 30428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:42.537949 30435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:42.541231 30435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 18:46:42 dist-test-slave-8tg2 krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:42.547657 30435 catalog_manager.cc:1347] Generated new cluster ID: e557c6cfb10547b590f29679e898c17c
I20260927 18:46:42.547719 30435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:42.550066 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:42.526092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.29.159.126:44083)
0927 18:46:42.526543 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:42.526562 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:42.527849 (+ 1287us) server_negotiation.cc:407] Connection header received
0927 18:46:42.531586 (+ 3737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:42.531599 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:42.531941 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:42.532336 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:42.535219 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:42.536902 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:42.538670 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:42.539128 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:42.539248 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:42.544306 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:42.544313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:42.544328 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:42.544358 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:42.546225 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:42.546869 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:42.546875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:42.546881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:42.547029 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:42.547491 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:42.547495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:42.547497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:42.548026 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:42.548240 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:42.548852 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 18:46:42.549142 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260927 18:46:42.588778 30435 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:42.589763 30435 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:42.602038 30435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95661d1e5e1d49539c42674b71d21421: Generated new TSK 0
I20260927 18:46:42.602932 30435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:46:42.635259 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44083
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:42.749851 30447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:42.750095 30447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:42.750171 30447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:42.754144 30447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:42.754202 30447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:42.754302 30447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:46:42.759411 30447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44083
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:42.760473 30447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:42.761451 30447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:42.770596 30453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:42.770619 30452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:42.770643 30447 server_base.cc:1034] running on GCE node
W20260927 18:46:42.770588 30455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:42.771227 30447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:42.771885 30447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:42.773039 30447 hybrid_clock.cc:648] HybridClock initialized: now 1790534802772985 us; error 67 us; skew 500 ppm
Sep 27 18:46:42 dist-test-slave-8tg2 krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534802, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:42.775743 30447 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:46:42.776520 30447 webserver.cc:460] Webserver started at http://127.29.159.65:46757/ using document root <none> and password file <none>
I20260927 18:46:42.777045 30447 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:42.777101 30447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:42.777344 30447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:42.779260 30447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "ae1870659bc74ef6ae3ac80b6b0aa01f"
format_stamp: "Formatted at 2026-09-27 18:46:42 on dist-test-slave-8tg2"
server_key: "b9216e6b8d9306b6a4f2c49df27cbe73"
server_key_iv: "fe3d04e790f4a7755af4907b5d0f4277"
server_key_version: "encryptionkey@0"
I20260927 18:46:42.779844 30447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "ae1870659bc74ef6ae3ac80b6b0aa01f"
format_stamp: "Formatted at 2026-09-27 18:46:42 on dist-test-slave-8tg2"
server_key: "b9216e6b8d9306b6a4f2c49df27cbe73"
server_key_iv: "fe3d04e790f4a7755af4907b5d0f4277"
server_key_version: "encryptionkey@0"
I20260927 18:46:42.783409 30447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:42.785633 30462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:42.786770 30447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 18:46:42.786891 30447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "ae1870659bc74ef6ae3ac80b6b0aa01f"
format_stamp: "Formatted at 2026-09-27 18:46:42 on dist-test-slave-8tg2"
server_key: "b9216e6b8d9306b6a4f2c49df27cbe73"
server_key_iv: "fe3d04e790f4a7755af4907b5d0f4277"
server_key_version: "encryptionkey@0"
I20260927 18:46:42.786991 30447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:42.823619 30447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:42.826792 30447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:42.827003 30447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:42.827776 30447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:42.828722 30447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:42.828794 30447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:42.828855 30447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:42.828902 30447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:42.838409 30447 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:32833
I20260927 18:46:42.838433 30575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:32833 every 8 connection(s)
I20260927 18:46:42.839334 30447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:42.842218 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 30447
I20260927 18:46:42.842350 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:42.842608 30333 external_mini_cluster.cc:1428] Setting key 930b4441a7b92c9c8ed8eeb7d8569459
Sep 27 18:46:42 dist-test-slave-8tg2 krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534802, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:46:42.853915 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:42.841262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:57257 (local address 127.29.159.126:44083)
0927 18:46:42.841411 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:42.841415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:42.842681 (+ 1266us) server_negotiation.cc:407] Connection header received
0927 18:46:42.843738 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:42.843740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:42.843784 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:42.843863 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:42.845231 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:42.845776 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:42.847087 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:42.847342 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:42.847478 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:42.850271 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:42.850275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:42.850277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:42.850300 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:42.851886 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:42.852420 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:42.852425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:42.852427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:42.852514 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:42.852843 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:42.852848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:42.852849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:42.853032 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:42.853148 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:42.853617 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 18:46:42.853748 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
I20260927 18:46:42.854756 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:42.841611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44083 (local address 127.29.159.65:57257)
0927 18:46:42.842524 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:42.842576 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:42.843509 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:42.844089 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:42.844097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:42.844574 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:42.845076 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:42.845089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:42.845905 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:42.845909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:42.846878 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:42.846887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:42.847508 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:42.847512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:42.847710 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:42.848325 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:42.848340 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:42.850094 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:42.852014 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:42.852020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:42.852035 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:42.852316 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:42.852603 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:42.852608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:42.852610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:42.852749 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:42.853135 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:42.853142 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:42.853390 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 18:46:42.853576 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 18:46:42.853902 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":241,"threads_started":1}
I20260927 18:46:42.856088 30576 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44083
I20260927 18:46:42.856365 30576 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:42.856967 30576 heartbeater.cc:511] Master 127.29.159.126:44083 requested a full tablet report, sending...
I20260927 18:46:42.858736 30385 ts_manager.cc:194] Registered new tserver with Master: ae1870659bc74ef6ae3ac80b6b0aa01f (127.29.159.65:32833)
I20260927 18:46:42.859931 30385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:57257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:46:42.871217 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44083
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:42.981215 30581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:42.981451 30581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:42.981494 30581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:42.985617 30581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:42.985677 30581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:42.985764 30581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:46:42.991087 30581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44083
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:42.992138 30581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:42.993136 30581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:43.002810 30587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:43.002858 30589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:43.002827 30586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:43.003120 30581 server_base.cc:1034] running on GCE node
I20260927 18:46:43.003829 30581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:43.004518 30581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:43.005694 30581 hybrid_clock.cc:648] HybridClock initialized: now 1790534803005663 us; error 53 us; skew 500 ppm
Sep 27 18:46:43 dist-test-slave-8tg2 krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534803, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:43.008515 30581 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:46:43.009389 30581 webserver.cc:460] Webserver started at http://127.29.159.66:42931/ using document root <none> and password file <none>
I20260927 18:46:43.009950 30581 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:43.010037 30581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:43.010259 30581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:43.012419 30581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "01a2102d44ad4c4699cbba9093c79310"
format_stamp: "Formatted at 2026-09-27 18:46:43 on dist-test-slave-8tg2"
server_key: "49b6316153067abb6780c06ff88ca3ca"
server_key_iv: "3e15dd7ffdb00cae87e2406bd42b72ce"
server_key_version: "encryptionkey@0"
I20260927 18:46:43.012990 30581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "01a2102d44ad4c4699cbba9093c79310"
format_stamp: "Formatted at 2026-09-27 18:46:43 on dist-test-slave-8tg2"
server_key: "49b6316153067abb6780c06ff88ca3ca"
server_key_iv: "3e15dd7ffdb00cae87e2406bd42b72ce"
server_key_version: "encryptionkey@0"
I20260927 18:46:43.016638 30581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:43.018988 30596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:43.020243 30581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:43.020378 30581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "01a2102d44ad4c4699cbba9093c79310"
format_stamp: "Formatted at 2026-09-27 18:46:43 on dist-test-slave-8tg2"
server_key: "49b6316153067abb6780c06ff88ca3ca"
server_key_iv: "3e15dd7ffdb00cae87e2406bd42b72ce"
server_key_version: "encryptionkey@0"
I20260927 18:46:43.020486 30581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:43.034997 30581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:43.038414 30581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:43.038601 30581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:43.039175 30581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:43.040022 30581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:43.040087 30581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:43.040156 30581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:43.040205 30581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:43.048935 30581 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:37075
I20260927 18:46:43.048959 30709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:37075 every 8 connection(s)
I20260927 18:46:43.049914 30581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:43.057443 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 30581
I20260927 18:46:43.057559 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:43.057826 30333 external_mini_cluster.cc:1428] Setting key 639c1b4b792c50914daaea45d2a689e0
Sep 27 18:46:43 dist-test-slave-8tg2 krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534803, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:43.063481 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:43.051817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:39741 (local address 127.29.159.126:44083)
0927 18:46:43.051953 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:43.051956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:43.052936 (+ 980us) server_negotiation.cc:407] Connection header received
0927 18:46:43.053777 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:43.053780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:43.053826 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:43.053902 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:43.055043 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.055689 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.056833 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.057045 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.057228 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:43.059935 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:43.059938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:43.059940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:43.059963 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:43.061390 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:43.061901 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:43.061906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:43.061908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:43.061984 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:43.062312 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:43.062316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:43.062318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:43.062491 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:43.062597 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:43.063161 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 18:46:43.063280 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:46:43.064479 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:43.052192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44083 (local address 127.29.159.66:39741)
0927 18:46:43.052810 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:43.052851 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:43.053598 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:43.054056 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:43.054063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:43.054429 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:43.054887 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:43.054899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.055831 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.055834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:43.056715 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:43.056724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.057228 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.057232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:43.057370 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:43.058065 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:43.058081 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:43.059755 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:43.061523 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:43.061529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:43.061542 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:43.061793 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:43.062078 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:43.062081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:43.062083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:43.062217 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:43.062595 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:43.062602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:43.062936 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 18:46:43.063120 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 18:46:43.063431 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":116,"threads_started":1}
I20260927 18:46:43.065603 30710 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44083
I20260927 18:46:43.065840 30710 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:43.066370 30710 heartbeater.cc:511] Master 127.29.159.126:44083 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:46:43.067590 30385 ts_manager.cc:194] Registered new tserver with Master: 01a2102d44ad4c4699cbba9093c79310 (127.29.159.66:37075)
I20260927 18:46:43.068264 30385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:39741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:46:43.086685 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44083
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:43.196949 30715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:43.197253 30715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:43.197343 30715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:43.201407 30715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:43.201495 30715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:43.201586 30715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:46:43.206935 30715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44083
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:43.208057 30715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:43.209085 30715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:43.218655 30720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:43.218665 30721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:43.218792 30723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:43.218972 30715 server_base.cc:1034] running on GCE node
I20260927 18:46:43.219486 30715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:43.220171 30715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:43.221323 30715 hybrid_clock.cc:648] HybridClock initialized: now 1790534803221289 us; error 48 us; skew 500 ppm
Sep 27 18:46:43 dist-test-slave-8tg2 krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534803, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:43.224181 30715 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:46:43.225092 30715 webserver.cc:460] Webserver started at http://127.29.159.67:44093/ using document root <none> and password file <none>
I20260927 18:46:43.225684 30715 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:43.225745 30715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:43.226085 30715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:43.228348 30715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "759dd753448f4da8b31da510e86deafa"
format_stamp: "Formatted at 2026-09-27 18:46:43 on dist-test-slave-8tg2"
server_key: "ada81e5b4b0caff3fb2b20cde2c79658"
server_key_iv: "c1f06d30941e57de2709adc0550ab00e"
server_key_version: "encryptionkey@0"
I20260927 18:46:43.228948 30715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "759dd753448f4da8b31da510e86deafa"
format_stamp: "Formatted at 2026-09-27 18:46:43 on dist-test-slave-8tg2"
server_key: "ada81e5b4b0caff3fb2b20cde2c79658"
server_key_iv: "c1f06d30941e57de2709adc0550ab00e"
server_key_version: "encryptionkey@0"
I20260927 18:46:43.232942 30715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:46:43.235528 30730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:43.237015 30715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:46:43.237133 30715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "759dd753448f4da8b31da510e86deafa"
format_stamp: "Formatted at 2026-09-27 18:46:43 on dist-test-slave-8tg2"
server_key: "ada81e5b4b0caff3fb2b20cde2c79658"
server_key_iv: "c1f06d30941e57de2709adc0550ab00e"
server_key_version: "encryptionkey@0"
I20260927 18:46:43.237259 30715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:43.252221 30715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:43.255110 30715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:43.255317 30715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:43.255963 30715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:43.256790 30715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:43.256837 30715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:43.256901 30715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:43.256935 30715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:43.266047 30715 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:44551
I20260927 18:46:43.266072 30843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:44551 every 8 connection(s)
I20260927 18:46:43.267053 30715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:43.272904 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 30715
I20260927 18:46:43.273005 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:43.273320 30333 external_mini_cluster.cc:1428] Setting key 87823471612685d9d1010ae7c8edbc72
Sep 27 18:46:43 dist-test-slave-8tg2 krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534803, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:43.282326 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:43.268905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:38755 (local address 127.29.159.126:44083)
0927 18:46:43.269043 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:43.269046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:43.270068 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 18:46:43.271384 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:43.271387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:43.271473 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:43.271580 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:43.272817 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.273463 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.274820 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.275097 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.275191 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:43.277901 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:43.277904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:43.277906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:43.277930 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:43.279575 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:43.280231 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:43.280235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:43.280237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:43.280344 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:43.280779 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:43.280784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:43.280788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:43.281087 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:43.281188 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:43.282027 (+ 839us) server_negotiation.cc:299] Negotiation successful
0927 18:46:43.282154 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:46:43.283165 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:43.269305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44083 (local address 127.29.159.67:38755)
0927 18:46:43.269912 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:43.269956 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:43.271157 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:43.271760 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:43.271768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:43.272145 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:43.272665 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:43.272680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.273585 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.273593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:43.274696 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:43.274706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.275287 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.275290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:43.275494 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:43.276175 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:43.276197 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:43.277747 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:43.279696 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:43.279702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:43.279718 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:43.280092 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:43.280466 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:43.280471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:43.280473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:43.280673 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:43.281209 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:43.281216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:43.281659 (+ 443us) client_negotiation.cc:769] Sending connection context
0927 18:46:43.281938 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 18:46:43.282283 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":278,"threads_started":1}
I20260927 18:46:43.284793 30844 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44083
I20260927 18:46:43.285027 30844 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:43.285511 30844 heartbeater.cc:511] Master 127.29.159.126:44083 requested a full tablet report, sending...
I20260927 18:46:43.286597 30385 ts_manager.cc:194] Registered new tserver with Master: 759dd753448f4da8b31da510e86deafa (127.29.159.67:44551)
I20260927 18:46:43.287297 30385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:38755
I20260927 18:46:43.288005 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:46:43 dist-test-slave-8tg2 krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:46:43 dist-test-slave-8tg2 krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:46:43.314301 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:43.304787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.29.159.65:32833)
0927 18:46:43.305101 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:43.305118 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:43.305146 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:46:43.305353 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:43.305365 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:43.305557 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:43.305715 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:43.306383 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.307187 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.308065 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.308461 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.308585 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:43.310971 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:43.310979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:43.310992 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:43.311015 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:43.312550 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:43.312990 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:43.312995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:43.312997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:43.313076 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:43.313347 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:43.313351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:43.313353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:43.313583 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:43.313822 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:43.313973 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 18:46:43.314124 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":103,"threads_started":1}
W20260927 18:46:43.315277 30490 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42944
Sep 27 18:46:43 dist-test-slave-8tg2 krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:46:43 dist-test-slave-8tg2 krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:46:43.341622 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:43.333277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.29.159.65:32833)
0927 18:46:43.333444 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:43.333448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:43.333491 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:46:43.333626 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:43.333629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:43.333671 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:43.333740 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:43.334454 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.334986 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.335927 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:43.336150 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:43.336268 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:43.338691 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:43.338694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:43.338696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:43.338718 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:43.339999 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:43.340447 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:43.340449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:43.340450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:43.340519 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:43.340755 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:43.340757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:43.340758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:43.340965 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:43.341065 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:43.341320 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 18:46:43.341415 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:46:43.342646 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 30447
I20260927 18:46:43.349385 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 30581
I20260927 18:46:43.354547 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 30715
I20260927 18:46:43.359702 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 30353
2026-09-27T18:46:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2211 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:43 dist-test-slave-8tg2 krb5kdc[30858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:43 dist-test-slave-8tg2 krb5kdc[30858](info): set up 2 sockets
Sep 27 18:46:43 dist-test-slave-8tg2 krb5kdc[30858](info): commencing operation
krb5kdc: starting...
W20260927 18:46:44.411167 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:46:44 dist-test-slave-8tg2 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:46:44.530740 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:43397
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:43397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:44.640122 30871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:44.640421 30871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:44.640460 30871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:44.644948 30871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:44.645013 30871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:44.645033 30871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:44.645049 30871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:44.645064 30871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:44.650241 30871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:43397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:43397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:44.651268 30871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:44.652271 30871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:44.660491 30877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:44.660491 30879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:44.660548 30876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:44.660809 30871 server_base.cc:1034] running on GCE node
I20260927 18:46:44.661505 30871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:44.662556 30871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:44.663710 30871 hybrid_clock.cc:648] HybridClock initialized: now 1790534804663690 us; error 32 us; skew 500 ppm
Sep 27 18:46:44 dist-test-slave-8tg2 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534804, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:44.673556 30871 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:46:44.674541 30871 webserver.cc:460] Webserver started at http://127.29.159.126:38889/ using document root <none> and password file <none>
I20260927 18:46:44.675117 30871 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:44.675204 30871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:44.675493 30871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:44.677582 30871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "9749a1e01da94621aba9260317a13652"
format_stamp: "Formatted at 2026-09-27 18:46:44 on dist-test-slave-8tg2"
server_key: "9321f65c50818b677b584d0cd13f99e7"
server_key_iv: "3085b7e3fd23018e650473a75bdafae5"
server_key_version: "encryptionkey@0"
I20260927 18:46:44.678153 30871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "9749a1e01da94621aba9260317a13652"
format_stamp: "Formatted at 2026-09-27 18:46:44 on dist-test-slave-8tg2"
server_key: "9321f65c50818b677b584d0cd13f99e7"
server_key_iv: "3085b7e3fd23018e650473a75bdafae5"
server_key_version: "encryptionkey@0"
I20260927 18:46:44.681977 30871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 18:46:44.684424 30886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:44.685694 30871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:44.685825 30871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "9749a1e01da94621aba9260317a13652"
format_stamp: "Formatted at 2026-09-27 18:46:44 on dist-test-slave-8tg2"
server_key: "9321f65c50818b677b584d0cd13f99e7"
server_key_iv: "3085b7e3fd23018e650473a75bdafae5"
server_key_version: "encryptionkey@0"
I20260927 18:46:44.685935 30871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:44.720523 30871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:44.724521 30871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:44.724774 30871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:44.732471 30871 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:43397
I20260927 18:46:44.732477 30938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:43397 every 8 connection(s)
I20260927 18:46:44.733522 30871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:44.736400 30939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:44.737071 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 30871
I20260927 18:46:44.737219 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:46:44.737509 30333 external_mini_cluster.cc:1428] Setting key b90bdc767aaba14d51726726fb15b3cd
I20260927 18:46:44.742612 30939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652: Bootstrap starting.
I20260927 18:46:44.745189 30939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:44.746129 30939 log.cc:824] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:46:44 dist-test-slave-8tg2 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:44.748328 30939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652: No bootstrap required, opened a new log
I20260927 18:46:44.751729 30939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9749a1e01da94621aba9260317a13652" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43397 } }
I20260927 18:46:44.752012 30939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:44.752084 30939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9749a1e01da94621aba9260317a13652, State: Initialized, Role: FOLLOWER
I20260927 18:46:44.752530 30939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9749a1e01da94621aba9260317a13652" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43397 } }
I20260927 18:46:44.752650 30939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:44.752725 30939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:44.752832 30939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:44.752943 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:44.738942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.29.159.126:43397)
0927 18:46:44.739390 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:44.739407 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:44.739495 (+ 88us) server_negotiation.cc:407] Connection header received
0927 18:46:44.740165 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:44.740183 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:44.740536 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:44.740839 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:44.741716 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:44.742934 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:44.744123 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:44.744652 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:44.744784 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:44.747968 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:44.747973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:44.747987 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:44.748015 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:44.749848 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:44.750309 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:44.750320 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:44.750325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:44.750440 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:44.750746 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:44.750752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:44.750754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:44.751157 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:44.751383 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:44.751686 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 18:46:44.751981 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260927 18:46:44.753713 30939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9749a1e01da94621aba9260317a13652" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43397 } }
I20260927 18:46:44.754034 30939 leader_election.cc:302] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9749a1e01da94621aba9260317a13652; no voters:
I20260927 18:46:44.754349 30939 leader_election.cc:288] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:44.754415 30944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:44.754649 30944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [term 1 LEADER]: Becoming Leader. State: Replica: 9749a1e01da94621aba9260317a13652, State: Running, Role: LEADER
I20260927 18:46:44.754941 30944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9749a1e01da94621aba9260317a13652" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43397 } }
I20260927 18:46:44.755647 30939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:44.756105 30944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9749a1e01da94621aba9260317a13652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9749a1e01da94621aba9260317a13652" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43397 } } }
I20260927 18:46:44.756217 30944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:44.756556 30945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9749a1e01da94621aba9260317a13652. Latest consensus state: current_term: 1 leader_uuid: "9749a1e01da94621aba9260317a13652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9749a1e01da94621aba9260317a13652" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43397 } } }
I20260927 18:46:44.756690 30945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652 [sys.catalog]: This master's current role is: LEADER
W20260927 18:46:44.760315 30959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:46:44.760376 30959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:46:44.760452 30949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:44.761221 30949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:44.767343 30949 catalog_manager.cc:1347] Generated new cluster ID: efc2dc0e34844027815c37235d99a956
I20260927 18:46:44.767411 30949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:44.785280 30949 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:44.786710 30949 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:44.799125 30949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9749a1e01da94621aba9260317a13652: Generated new TSK 0
I20260927 18:46:44.800132 30949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:46:44.834609 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43397
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:44.948560 30965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:44.948865 30965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:44.948912 30965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:44.953084 30965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:44.953143 30965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:44.953205 30965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:46:44.958707 30965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:44.959872 30965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:44.960978 30965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:44.970711 30973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:44.970774 30965 server_base.cc:1034] running on GCE node
W20260927 18:46:44.970928 30971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:44.970925 30970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:44.971386 30965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:44.972069 30965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:44.973249 30965 hybrid_clock.cc:648] HybridClock initialized: now 1790534804973232 us; error 29 us; skew 500 ppm
Sep 27 18:46:44 dist-test-slave-8tg2 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534804, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:44.975998 30965 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:46:44.977025 30965 webserver.cc:460] Webserver started at http://127.29.159.65:37395/ using document root <none> and password file <none>
I20260927 18:46:44.977546 30965 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:44.977604 30965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:44.977770 30965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:44.979892 30965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "c957ef9fd4c04f1ea70a65b337505c19"
format_stamp: "Formatted at 2026-09-27 18:46:44 on dist-test-slave-8tg2"
server_key: "5baf5488d3abd6063606527ac517c647"
server_key_iv: "be2d012aa9c24a20d6a61f64ef3f3389"
server_key_version: "encryptionkey@0"
I20260927 18:46:44.980407 30965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "c957ef9fd4c04f1ea70a65b337505c19"
format_stamp: "Formatted at 2026-09-27 18:46:44 on dist-test-slave-8tg2"
server_key: "5baf5488d3abd6063606527ac517c647"
server_key_iv: "be2d012aa9c24a20d6a61f64ef3f3389"
server_key_version: "encryptionkey@0"
I20260927 18:46:44.984159 30965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:46:44.986454 30980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:44.987702 30965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:44.987830 30965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "c957ef9fd4c04f1ea70a65b337505c19"
format_stamp: "Formatted at 2026-09-27 18:46:44 on dist-test-slave-8tg2"
server_key: "5baf5488d3abd6063606527ac517c647"
server_key_iv: "be2d012aa9c24a20d6a61f64ef3f3389"
server_key_version: "encryptionkey@0"
I20260927 18:46:44.987936 30965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:45.001803 30965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:45.004827 30965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:45.005026 30965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:45.005636 30965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:45.006444 30965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:45.006490 30965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:45.006556 30965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:45.006585 30965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:45.015024 30965 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:40717
I20260927 18:46:45.015110 31093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:40717 every 8 connection(s)
I20260927 18:46:45.015959 30965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:45.021642 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 30965
I20260927 18:46:45.021763 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:45.022068 30333 external_mini_cluster.cc:1428] Setting key 71857ea2f981fc2c1c2c7850ef3dec6d
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534804, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:45.030948 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.017722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:43751 (local address 127.29.159.126:43397)
0927 18:46:45.017850 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.017853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.018730 (+ 877us) server_negotiation.cc:407] Connection header received
0927 18:46:45.020278 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.020281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.020340 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.020450 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:45.021866 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.022655 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.024245 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.024544 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.024642 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.027387 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:45.027390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:45.027392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:45.027415 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:45.028949 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.029478 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.029482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.029484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.029561 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.029884 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.029889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.029891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.030061 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:45.030162 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.030639 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.030764 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:46:45.031777 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.018089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:43397 (local address 127.29.159.65:43751)
0927 18:46:45.018563 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:45.018609 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:45.020003 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:45.020629 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:45.020639 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:45.021010 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:45.021661 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.021685 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.022798 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.022801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.024084 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.024099 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.024704 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.024711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.024972 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.025699 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:45.025716 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:45.027210 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:45.029060 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:45.029066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:45.029080 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:45.029376 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:45.029657 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:45.029660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:45.029661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:45.029794 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:45.030161 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:45.030167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:45.030410 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 18:46:45.030587 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 18:46:45.030893 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
I20260927 18:46:45.032948 31094 heartbeater.cc:347] Connected to a master server at 127.29.159.126:43397
I20260927 18:46:45.033201 31094 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:45.033764 31094 heartbeater.cc:511] Master 127.29.159.126:43397 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
I20260927 18:46:45.035352 30903 ts_manager.cc:194] Registered new tserver with Master: c957ef9fd4c04f1ea70a65b337505c19 (127.29.159.65:40717)
I20260927 18:46:45.036563 30903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:43751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:46:45.050225 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43397
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:45.162173 31099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:45.162460 31099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:45.162506 31099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:45.166824 31099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:45.166908 31099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:45.166996 31099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:46:45.172550 31099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:45.173563 31099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:45.174623 31099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:45.184432 31107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:45.184557 31099 server_base.cc:1034] running on GCE node
W20260927 18:46:45.184401 31105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:45.184412 31104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:45.185060 31099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:45.185676 31099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:45.186847 31099 hybrid_clock.cc:648] HybridClock initialized: now 1790534805186829 us; error 28 us; skew 500 ppm
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534805, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:45.189756 31099 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:46:45.190733 31099 webserver.cc:460] Webserver started at http://127.29.159.66:44759/ using document root <none> and password file <none>
I20260927 18:46:45.191295 31099 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:45.191385 31099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:45.191655 31099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:45.193718 31099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "bd86b32a5c364073b6ccd179f3056aff"
format_stamp: "Formatted at 2026-09-27 18:46:45 on dist-test-slave-8tg2"
server_key: "4bebd42606283c09813f6f276e57c091"
server_key_iv: "f5196f5ba03f4986383e89592dbf18b4"
server_key_version: "encryptionkey@0"
I20260927 18:46:45.194304 31099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "bd86b32a5c364073b6ccd179f3056aff"
format_stamp: "Formatted at 2026-09-27 18:46:45 on dist-test-slave-8tg2"
server_key: "4bebd42606283c09813f6f276e57c091"
server_key_iv: "f5196f5ba03f4986383e89592dbf18b4"
server_key_version: "encryptionkey@0"
I20260927 18:46:45.198048 31099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:46:45.200659 31114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:45.201895 31099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:46:45.202029 31099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "bd86b32a5c364073b6ccd179f3056aff"
format_stamp: "Formatted at 2026-09-27 18:46:45 on dist-test-slave-8tg2"
server_key: "4bebd42606283c09813f6f276e57c091"
server_key_iv: "f5196f5ba03f4986383e89592dbf18b4"
server_key_version: "encryptionkey@0"
I20260927 18:46:45.202144 31099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:45.226775 31099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:45.230690 31099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:45.230907 31099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:45.231546 31099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:45.232388 31099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:45.232463 31099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:45.232554 31099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:45.232597 31099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:45.241313 31099 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:40671
I20260927 18:46:45.241335 31227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:40671 every 8 connection(s)
I20260927 18:46:45.242190 31099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:45.246493 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 31099
I20260927 18:46:45.246637 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:45.246902 30333 external_mini_cluster.cc:1428] Setting key 61c1fe0c2c021623ab15450d447deabb
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534805, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:46:45.257018 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.244019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:55003 (local address 127.29.159.126:43397)
0927 18:46:45.244158 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.244161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.244979 (+ 818us) server_negotiation.cc:407] Connection header received
0927 18:46:45.246359 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.246363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.246426 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.246523 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:45.247972 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.248601 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.249722 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.249953 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.250061 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.252506 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:45.252509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:45.252511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:45.252535 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:45.254239 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.254770 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.254774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.254776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.254876 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.255213 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.255218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.255222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.255520 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:45.255645 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.256355 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.256810 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:46:45.257746 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.244391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:43397 (local address 127.29.159.66:55003)
0927 18:46:45.244830 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:45.244874 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:45.246107 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:45.246752 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:45.246760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:45.247209 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:45.247836 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.247850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.248721 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.248729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.249599 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.249608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.250098 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.250103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.250267 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.250901 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:45.250916 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:45.252335 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:45.254380 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:45.254386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:45.254400 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:45.254666 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:45.254983 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:45.254985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:45.254986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:45.255123 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:45.255689 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:45.255698 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:45.256037 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 18:46:45.256243 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 18:46:45.256594 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260927 18:46:45.259195 31228 heartbeater.cc:347] Connected to a master server at 127.29.159.126:43397
I20260927 18:46:45.259501 31228 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:46:45.260123 31228 heartbeater.cc:511] Master 127.29.159.126:43397 requested a full tablet report, sending...
I20260927 18:46:45.261305 30903 ts_manager.cc:194] Registered new tserver with Master: bd86b32a5c364073b6ccd179f3056aff (127.29.159.66:40671)
I20260927 18:46:45.261925 30903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:55003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:46:45.276638 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43397
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:45.388361 31233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:45.388659 31233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:45.388746 31233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:45.392925 31233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:45.392983 31233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:45.393043 31233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:46:45.398561 31233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:45.399631 31233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:45.400617 31233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:46:45.410555 31233 server_base.cc:1034] running on GCE node
W20260927 18:46:45.410516 31238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:45.410542 31239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:45.410516 31241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:45.411227 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:45.411949 31233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:45.413156 31233 hybrid_clock.cc:648] HybridClock initialized: now 1790534805413114 us; error 31 us; skew 500 ppm
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534805, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:45.415926 31233 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:46:45.416838 31233 webserver.cc:460] Webserver started at http://127.29.159.67:39027/ using document root <none> and password file <none>
I20260927 18:46:45.417414 31233 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:45.417502 31233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:45.417730 31233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:45.419883 31233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "2541928e930842fe9c0c6da57f1ea675"
format_stamp: "Formatted at 2026-09-27 18:46:45 on dist-test-slave-8tg2"
server_key: "89d0ed7772a93b99d91de3c427f2c6b7"
server_key_iv: "9b63c685999974747baedb1ae73b11c8"
server_key_version: "encryptionkey@0"
I20260927 18:46:45.420462 31233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "2541928e930842fe9c0c6da57f1ea675"
format_stamp: "Formatted at 2026-09-27 18:46:45 on dist-test-slave-8tg2"
server_key: "89d0ed7772a93b99d91de3c427f2c6b7"
server_key_iv: "9b63c685999974747baedb1ae73b11c8"
server_key_version: "encryptionkey@0"
I20260927 18:46:45.424360 31233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:46:45.426764 31248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:45.427994 31233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:45.428126 31233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "2541928e930842fe9c0c6da57f1ea675"
format_stamp: "Formatted at 2026-09-27 18:46:45 on dist-test-slave-8tg2"
server_key: "89d0ed7772a93b99d91de3c427f2c6b7"
server_key_iv: "9b63c685999974747baedb1ae73b11c8"
server_key_version: "encryptionkey@0"
I20260927 18:46:45.428242 31233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:45.440970 31233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:45.444006 31233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:45.444213 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:45.444792 31233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:45.445612 31233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:45.445683 31233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:45.445753 31233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:45.445801 31233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:45.454414 31233 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:45255
I20260927 18:46:45.454432 31361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:45255 every 8 connection(s)
I20260927 18:46:45.455343 31233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:45.462953 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 31233
I20260927 18:46:45.463052 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:45.463361 30333 external_mini_cluster.cc:1428] Setting key a3fac75d588311b3f337c9ee0dd8ec9d
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534805, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:45.469393 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.457241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:36067 (local address 127.29.159.126:43397)
0927 18:46:45.457414 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.457419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.458246 (+ 827us) server_negotiation.cc:407] Connection header received
0927 18:46:45.459904 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.459907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.459954 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.460028 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:45.461135 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.461663 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.462841 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.463114 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.463204 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.466126 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:45.466129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:45.466131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:45.466156 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:45.467616 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.468059 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.468061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.468062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.468132 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.468484 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.468486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.468487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.468653 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:45.468725 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.469172 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.469266 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:46:45.470213 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.457598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:43397 (local address 127.29.159.67:36067)
0927 18:46:45.458049 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:45.458091 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:45.459704 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:45.460190 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:45.460196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:45.460536 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:45.461003 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.461016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.461764 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.461766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.462722 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.462734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.463265 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.463268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.463501 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.464351 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:45.464372 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:45.465991 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:45.467709 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:45.467713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:45.467725 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:45.467978 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:45.468236 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:45.468238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:45.468240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:45.468398 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:45.468737 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:45.468742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:45.468974 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 18:46:45.469142 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 18:46:45.469388 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":137,"threads_started":1}
I20260927 18:46:45.471346 31362 heartbeater.cc:347] Connected to a master server at 127.29.159.126:43397
I20260927 18:46:45.471634 31362 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:45.472114 31362 heartbeater.cc:511] Master 127.29.159.126:43397 requested a full tablet report, sending...
I20260927 18:46:45.473126 30903 ts_manager.cc:194] Registered new tserver with Master: 2541928e930842fe9c0c6da57f1ea675 (127.29.159.67:45255)
I20260927 18:46:45.473688 30903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:36067
I20260927 18:46:45.477130 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:46:45.487381 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.480409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.29.159.126:43397)
0927 18:46:45.480528 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.480533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.480708 (+ 175us) server_negotiation.cc:407] Connection header received
0927 18:46:45.480874 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.480878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.480921 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.480998 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:45.481643 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.482139 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.482965 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.483183 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.483283 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.484591 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:45.484599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:45.484601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:45.484622 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:45.485773 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.486185 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.486190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.486192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.486267 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.486533 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.486538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.486539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.486731 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:45.486889 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.487111 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.487222 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:46:45.491937 30903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:45.494871 30903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:45.508790 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.502817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.29.159.67:45255)
0927 18:46:45.503476 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.503491 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.503835 (+ 344us) server_negotiation.cc:407] Connection header received
0927 18:46:45.503922 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.503927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.504139 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.504300 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:45.504632 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.505673 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.507309 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.508143 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.508257 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.508400 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.508536 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.508626 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":142,"threads_started":1}
I20260927 18:46:45.508821 31373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.502683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:45255 (local address 127.0.0.1:37660)
0927 18:46:45.503498 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:45.503566 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:45.503703 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:45.504324 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:45.504332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:45.504358 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:45.504511 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.504519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.505793 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.505799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.507197 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.507207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.508257 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.508261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.508367 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.508403 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:46:45.508510 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:46:45.508628 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":66,"thread_start_us":102,"threads_started":1}
I20260927 18:46:45.509178 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.503022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.29.159.66:40671)
0927 18:46:45.503403 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.503415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.503933 (+ 518us) server_negotiation.cc:407] Connection header received
0927 18:46:45.504124 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.504129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.504302 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.504423 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:45.504848 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.506039 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.507527 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.508116 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.508221 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.508359 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.508948 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.509045 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":95,"threads_started":1}
I20260927 18:46:45.508848 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.502862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:40671 (local address 127.0.0.1:55090)
0927 18:46:45.503551 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:45.503567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:45.503991 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:45.504595 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:45.504598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:45.504615 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:45.504726 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.504733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.506181 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.506184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.507389 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.507396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.508422 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.508425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.508515 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.508538 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:46:45.508606 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:46:45.508690 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":52,"thread_start_us":96,"threads_started":1}
I20260927 18:46:45.509603 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.503591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:40717 (local address 127.0.0.1:49082)
0927 18:46:45.503871 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:45.503884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:45.503971 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:45.505141 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:45.505145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:45.505164 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:45.505255 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.505261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.506689 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.506692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.507940 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.507951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.509189 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.509195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.509294 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.509318 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:46:45.509395 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:46:45.509484 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":38,"thread_start_us":87,"threads_started":1}
I20260927 18:46:45.509980 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.503891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.29.159.65:40717)
0927 18:46:45.504640 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.504654 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.504680 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:46:45.504748 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.504751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.504954 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.505111 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:45.505356 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.506518 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.508078 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.508894 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.508996 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.509110 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.509757 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.509852 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":123,"threads_started":1}
I20260927 18:46:45.512081 31296 tablet_service.cc:1467] Processing CreateTablet for tablet ebd852e67bd64675bbc29c1c2762dbc8 (DEFAULT_TABLE table=test-table [id=bf5e3880fe754bc3a157930d581a50e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:45.512128 31028 tablet_service.cc:1467] Processing CreateTablet for tablet ebd852e67bd64675bbc29c1c2762dbc8 (DEFAULT_TABLE table=test-table [id=bf5e3880fe754bc3a157930d581a50e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:45.512128 31162 tablet_service.cc:1467] Processing CreateTablet for tablet ebd852e67bd64675bbc29c1c2762dbc8 (DEFAULT_TABLE table=test-table [id=bf5e3880fe754bc3a157930d581a50e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:45.513057 31296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd852e67bd64675bbc29c1c2762dbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:45.513103 31162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd852e67bd64675bbc29c1c2762dbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:45.513103 31028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd852e67bd64675bbc29c1c2762dbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:45.519810 31379 tablet_bootstrap.cc:492] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675: Bootstrap starting.
I20260927 18:46:45.520452 31380 tablet_bootstrap.cc:492] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff: Bootstrap starting.
I20260927 18:46:45.521818 31379 tablet_bootstrap.cc:654] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:45.522396 31380 tablet_bootstrap.cc:654] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:45.522886 31379 log.cc:824] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:45.523087 31380 log.cc:824] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:45.523618 31381 tablet_bootstrap.cc:492] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19: Bootstrap starting.
I20260927 18:46:45.524999 31379 tablet_bootstrap.cc:492] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675: No bootstrap required, opened a new log
I20260927 18:46:45.524997 31380 tablet_bootstrap.cc:492] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff: No bootstrap required, opened a new log
I20260927 18:46:45.525192 31379 ts_tablet_manager.cc:1397] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:46:45.525192 31380 ts_tablet_manager.cc:1397] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:46:45.525717 31381 tablet_bootstrap.cc:654] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:45.526737 31381 log.cc:824] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:45.528474 31381 tablet_bootstrap.cc:492] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19: No bootstrap required, opened a new log
I20260927 18:46:45.528673 31381 ts_tablet_manager.cc:1397] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:45.529189 31379 raft_consensus.cc:348] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2541928e930842fe9c0c6da57f1ea675" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45255 } } peers { permanent_uuid: "bd86b32a5c364073b6ccd179f3056aff" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 40671 } } peers { permanent_uuid: "c957ef9fd4c04f1ea70a65b337505c19" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40717 } }
I20260927 18:46:45.529189 31380 raft_consensus.cc:348] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2541928e930842fe9c0c6da57f1ea675" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45255 } } peers { permanent_uuid: "bd86b32a5c364073b6ccd179f3056aff" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 40671 } } peers { permanent_uuid: "c957ef9fd4c04f1ea70a65b337505c19" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40717 } }
I20260927 18:46:45.529477 31380 raft_consensus.cc:374] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:45.529477 31379 raft_consensus.cc:374] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:45.529537 31379 raft_consensus.cc:729] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2541928e930842fe9c0c6da57f1ea675, State: Initialized, Role: FOLLOWER
I20260927 18:46:45.529537 31380 raft_consensus.cc:729] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd86b32a5c364073b6ccd179f3056aff, State: Initialized, Role: FOLLOWER
I20260927 18:46:45.530004 31380 consensus_queue.cc:260] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2541928e930842fe9c0c6da57f1ea675" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45255 } } peers { permanent_uuid: "bd86b32a5c364073b6ccd179f3056aff" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 40671 } } peers { permanent_uuid: "c957ef9fd4c04f1ea70a65b337505c19" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40717 } }
I20260927 18:46:45.530004 31379 consensus_queue.cc:260] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2541928e930842fe9c0c6da57f1ea675" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45255 } } peers { permanent_uuid: "bd86b32a5c364073b6ccd179f3056aff" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 40671 } } peers { permanent_uuid: "c957ef9fd4c04f1ea70a65b337505c19" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40717 } }
I20260927 18:46:45.530678 31362 heartbeater.cc:503] Master 127.29.159.126:43397 was elected leader, sending a full tablet report...
I20260927 18:46:45.530992 31380 ts_tablet_manager.cc:1428] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260927 18:46:45.530992 31379 ts_tablet_manager.cc:1428] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260927 18:46:45.531579 31228 heartbeater.cc:503] Master 127.29.159.126:43397 was elected leader, sending a full tablet report...
I20260927 18:46:45.534376 31381 raft_consensus.cc:348] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2541928e930842fe9c0c6da57f1ea675" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45255 } } peers { permanent_uuid: "bd86b32a5c364073b6ccd179f3056aff" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 40671 } } peers { permanent_uuid: "c957ef9fd4c04f1ea70a65b337505c19" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40717 } }
I20260927 18:46:45.534649 31381 raft_consensus.cc:374] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:45.534713 31381 raft_consensus.cc:729] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c957ef9fd4c04f1ea70a65b337505c19, State: Initialized, Role: FOLLOWER
I20260927 18:46:45.535138 31381 consensus_queue.cc:260] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2541928e930842fe9c0c6da57f1ea675" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45255 } } peers { permanent_uuid: "bd86b32a5c364073b6ccd179f3056aff" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 40671 } } peers { permanent_uuid: "c957ef9fd4c04f1ea70a65b337505c19" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40717 } }
I20260927 18:46:45.535707 31094 heartbeater.cc:503] Master 127.29.159.126:43397 was elected leader, sending a full tablet report...
I20260927 18:46:45.536062 31381 ts_tablet_manager.cc:1428] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 18:46:45.608371 31386 raft_consensus.cc:482] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:45.608553 31386 raft_consensus.cc:504] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2541928e930842fe9c0c6da57f1ea675" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45255 } } peers { permanent_uuid: "bd86b32a5c364073b6ccd179f3056aff" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 40671 } } peers { permanent_uuid: "c957ef9fd4c04f1ea70a65b337505c19" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40717 } }
I20260927 18:46:45.609613 31386 leader_election.cc:288] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2541928e930842fe9c0c6da57f1ea675 (127.29.159.67:45255), c957ef9fd4c04f1ea70a65b337505c19 (127.29.159.65:40717)
I20260927 18:46:45.613534 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.610004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:46813 (local address 127.29.159.67:45255)
0927 18:46:45.610153 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.610159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.610175 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:46:45.610234 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.610240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.610297 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.610379 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:45.610694 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.611325 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.612467 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.612967 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.613054 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.613092 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.613339 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.613409 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:46:45.613523 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.609846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:45255 (local address 127.29.159.66:46813)
0927 18:46:45.609987 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:45.610003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:45.610111 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:45.610427 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:45.610430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:45.610447 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:45.610574 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.610582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.611446 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.611450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.612339 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.612348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.613073 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.613077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.613182 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.613208 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:46:45.613289 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:46:45.613383 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 18:46:45.613823 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.610416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:44031 (local address 127.29.159.65:40717)
0927 18:46:45.610576 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.610580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.610619 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:46:45.610675 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.610679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.610729 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.610782 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:45.611062 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.611674 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.612777 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.613269 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.613378 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.613421 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.613603 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.613677 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:46:45.613981 31388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.610143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:40717 (local address 127.29.159.66:44031)
0927 18:46:45.610480 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:45.610493 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:45.610572 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:45.610811 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:45.610813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:45.610829 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:45.610953 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.610959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.611768 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.611771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.612657 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:45.612667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.613377 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.613380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:45.613473 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.613495 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:46:45.613576 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:46:45.613800 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":129,"threads_started":1}
I20260927 18:46:45.614215 31316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd852e67bd64675bbc29c1c2762dbc8" candidate_uuid: "bd86b32a5c364073b6ccd179f3056aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2541928e930842fe9c0c6da57f1ea675" is_pre_election: true
I20260927 18:46:45.614565 31316 raft_consensus.cc:2454] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd86b32a5c364073b6ccd179f3056aff in term 0.
I20260927 18:46:45.614593 31048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd852e67bd64675bbc29c1c2762dbc8" candidate_uuid: "bd86b32a5c364073b6ccd179f3056aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c957ef9fd4c04f1ea70a65b337505c19" is_pre_election: true
I20260927 18:46:45.614854 31048 raft_consensus.cc:2454] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd86b32a5c364073b6ccd179f3056aff in term 0.
I20260927 18:46:45.615150 31116 leader_election.cc:302] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2541928e930842fe9c0c6da57f1ea675, bd86b32a5c364073b6ccd179f3056aff; no voters:
I20260927 18:46:45.615468 31386 raft_consensus.cc:2781] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:45.615541 31386 raft_consensus.cc:482] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:45.615572 31386 raft_consensus.cc:3037] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:45.616671 31386 raft_consensus.cc:504] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2541928e930842fe9c0c6da57f1ea675" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45255 } } peers { permanent_uuid: "bd86b32a5c364073b6ccd179f3056aff" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 40671 } } peers { permanent_uuid: "c957ef9fd4c04f1ea70a65b337505c19" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40717 } }
I20260927 18:46:45.617029 31386 leader_election.cc:288] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [CANDIDATE]: Term 1 election: Requested vote from peers 2541928e930842fe9c0c6da57f1ea675 (127.29.159.67:45255), c957ef9fd4c04f1ea70a65b337505c19 (127.29.159.65:40717)
I20260927 18:46:45.617496 31316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd852e67bd64675bbc29c1c2762dbc8" candidate_uuid: "bd86b32a5c364073b6ccd179f3056aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2541928e930842fe9c0c6da57f1ea675"
I20260927 18:46:45.617612 31316 raft_consensus.cc:3037] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:45.617638 31048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd852e67bd64675bbc29c1c2762dbc8" candidate_uuid: "bd86b32a5c364073b6ccd179f3056aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c957ef9fd4c04f1ea70a65b337505c19"
I20260927 18:46:45.617777 31048 raft_consensus.cc:3037] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:45.618721 31316 raft_consensus.cc:2454] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd86b32a5c364073b6ccd179f3056aff in term 1.
I20260927 18:46:45.618963 31048 raft_consensus.cc:2454] T ebd852e67bd64675bbc29c1c2762dbc8 P c957ef9fd4c04f1ea70a65b337505c19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd86b32a5c364073b6ccd179f3056aff in term 1.
I20260927 18:46:45.619055 31116 leader_election.cc:302] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2541928e930842fe9c0c6da57f1ea675, bd86b32a5c364073b6ccd179f3056aff; no voters:
I20260927 18:46:45.619233 31386 raft_consensus.cc:2781] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:45.619535 31386 raft_consensus.cc:686] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [term 1 LEADER]: Becoming Leader. State: Replica: bd86b32a5c364073b6ccd179f3056aff, State: Running, Role: LEADER
I20260927 18:46:45.619867 31386 consensus_queue.cc:237] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2541928e930842fe9c0c6da57f1ea675" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45255 } } peers { permanent_uuid: "bd86b32a5c364073b6ccd179f3056aff" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 40671 } } peers { permanent_uuid: "c957ef9fd4c04f1ea70a65b337505c19" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40717 } }
I20260927 18:46:45.623314 30902 catalog_manager.cc:5495] T ebd852e67bd64675bbc29c1c2762dbc8 P bd86b32a5c364073b6ccd179f3056aff reported cstate change: term changed from 0 to 1, leader changed from <none> to bd86b32a5c364073b6ccd179f3056aff (127.29.159.66). New cstate: current_term: 1 leader_uuid: "bd86b32a5c364073b6ccd179f3056aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2541928e930842fe9c0c6da57f1ea675" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd86b32a5c364073b6ccd179f3056aff" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 40671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c957ef9fd4c04f1ea70a65b337505c19" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40717 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:46:45.661417 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.652881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.29.159.65:40717)
0927 18:46:45.653030 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.653034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.653207 (+ 173us) server_negotiation.cc:407] Connection header received
0927 18:46:45.653303 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.653307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.653356 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.653449 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:45.654105 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.654607 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.655477 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.655696 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.655799 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.658342 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:45.658353 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:45.658374 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:45.658397 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:45.659913 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.660290 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.660295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.660296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.660377 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.660594 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.660596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.660597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.660816 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:45.660980 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.661120 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.661233 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534805, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534805, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:46:45.686326 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.678568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.29.159.65:40717)
0927 18:46:45.678699 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.678713 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.678843 (+ 130us) server_negotiation.cc:407] Connection header received
0927 18:46:45.678996 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.678998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.679038 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.679107 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:45.679759 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.680262 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.681011 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.681218 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.681292 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.683628 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:45.683632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:45.683634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:45.683672 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:45.684914 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.685269 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.685272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.685273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.685341 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.685551 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.685552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.685553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.685691 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:45.685810 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.686023 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.686112 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260927 18:46:45.706373 31363 tablet.cc:2369] T ebd852e67bd64675bbc29c1c2762dbc8 P 2541928e930842fe9c0c6da57f1ea675: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:46:45.710552 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:45.702681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.29.159.65:40717)
0927 18:46:45.702815 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:45.702819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:45.702976 (+ 157us) server_negotiation.cc:407] Connection header received
0927 18:46:45.703084 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:45.703087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:45.703129 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:45.703198 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:45.703925 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.704417 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.705173 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:45.705383 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:45.705456 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:45.707999 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:45.708002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:45.708004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:45.708025 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:45.709213 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.709578 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.709581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.709582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.709650 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:45.709854 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:45.709856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:45.709857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:45.709994 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:45.710088 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:45.710268 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 18:46:45.710356 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 18:46:45.711872 31008 tablet_service.cc:2908] Rejecting scan request for tablet ebd852e67bd64675bbc29c1c2762dbc8: Uninitialized: clean time has not yet been initialized
I20260927 18:46:45.713745 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 30965
I20260927 18:46:45.719342 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 31099
I20260927 18:46:45.724664 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 31233
I20260927 18:46:45.729614 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 30871
2026-09-27T18:46:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2369 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[31403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[31403](info): set up 2 sockets
Sep 27 18:46:45 dist-test-slave-8tg2 krb5kdc[31403](info): commencing operation
krb5kdc: starting...
W20260927 18:46:46.763044 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:46:46 dist-test-slave-8tg2 krb5kdc[31403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:46:46.883378 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:45821
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:45821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:46.994656 31416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:46.994948 31416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:46.995016 31416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:46.998970 31416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:46.999053 31416 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:46:46.999087 31416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:46.999140 31416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:46.999190 31416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:46.999228 31416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:47.004444 31416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:45821
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:45821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:47.005519 31416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:47.006589 31416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:47.015118 31424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:47.015125 31422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:47.015125 31421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:47.015190 31416 server_base.cc:1034] running on GCE node
I20260927 18:46:47.015946 31416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:47.017019 31416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:47.018200 31416 hybrid_clock.cc:648] HybridClock initialized: now 1790534807018186 us; error 28 us; skew 500 ppm
Sep 27 18:46:47 dist-test-slave-8tg2 krb5kdc[31403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534807, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:47.020951 31416 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:46:47.021781 31416 webserver.cc:460] Webserver started at http://127.29.159.126:45971/ using document root <none> and password file <none>
I20260927 18:46:47.022328 31416 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:47.022413 31416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:47.022630 31416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:47.024626 31416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "f3a37401665e49028f25192242ce5a97"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "694e32552229e7780b605c0040b31c17"
server_key_iv: "ac98839b0786fe5d8560111caffec220"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.025168 31416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "f3a37401665e49028f25192242ce5a97"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "694e32552229e7780b605c0040b31c17"
server_key_iv: "ac98839b0786fe5d8560111caffec220"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.028633 31416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:47.030843 31431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:47.032061 31416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:47.032187 31416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "f3a37401665e49028f25192242ce5a97"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "694e32552229e7780b605c0040b31c17"
server_key_iv: "ac98839b0786fe5d8560111caffec220"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.032295 31416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:47.070847 31416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:47.074124 31416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:47.074337 31416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:47.082051 31416 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:45821
I20260927 18:46:47.082110 31493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:45821 every 8 connection(s)
I20260927 18:46:47.082999 31416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:47.085716 31494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:47.089751 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 31416
I20260927 18:46:47.089885 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:46:47.090132 30333 external_mini_cluster.cc:1428] Setting key 4364187f0803cd52214a762a6a99363d
I20260927 18:46:47.091957 31494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97: Bootstrap starting.
I20260927 18:46:47.094064 31494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:47.094784 31494 log.cc:824] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:47.096560 31494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97: No bootstrap required, opened a new log
Sep 27 18:46:47 dist-test-slave-8tg2 krb5kdc[31403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:47.100286 31494 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a37401665e49028f25192242ce5a97" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45821 } }
I20260927 18:46:47.100528 31494 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:47.100605 31494 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a37401665e49028f25192242ce5a97, State: Initialized, Role: FOLLOWER
I20260927 18:46:47.101120 31494 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a37401665e49028f25192242ce5a97" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45821 } }
I20260927 18:46:47.101295 31494 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:47.101382 31494 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:47.101480 31494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:47.102322 31494 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a37401665e49028f25192242ce5a97" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45821 } }
I20260927 18:46:47.102680 31494 leader_election.cc:302] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3a37401665e49028f25192242ce5a97; no voters:
I20260927 18:46:47.103024 31494 leader_election.cc:288] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:47.103070 31499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:47.103295 31499 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [term 1 LEADER]: Becoming Leader. State: Replica: f3a37401665e49028f25192242ce5a97, State: Running, Role: LEADER
I20260927 18:46:47.103608 31499 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a37401665e49028f25192242ce5a97" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45821 } }
I20260927 18:46:47.103778 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.091348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.29.159.126:45821)
0927 18:46:47.091743 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.091759 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.091797 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:46:47.092460 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.092477 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.092798 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.093115 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:47.093876 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.094895 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.095822 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.096289 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.096456 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.098880 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:47.098885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:47.098898 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:47.098923 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:47.100597 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.101117 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.101121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.101125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.101229 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.101566 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.101574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.101576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.101942 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:47.102145 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:47.102434 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 18:46:47.102740 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260927 18:46:47.104285 31494 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:47.105185 31500 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3a37401665e49028f25192242ce5a97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a37401665e49028f25192242ce5a97" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45821 } } }
I20260927 18:46:47.105331 31500 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:47.105648 31501 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3a37401665e49028f25192242ce5a97. Latest consensus state: current_term: 1 leader_uuid: "f3a37401665e49028f25192242ce5a97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3a37401665e49028f25192242ce5a97" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45821 } } }
I20260927 18:46:47.105748 31501 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:47.105690 31508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:47.109129 31508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:47.114651 31508 catalog_manager.cc:1347] Generated new cluster ID: 109e9d8b55fa46448ac9ce662d677f12
I20260927 18:46:47.114722 31508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:47.139446 31508 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:47.140746 31508 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:47.154186 31508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97: Generated new TSK 0
I20260927 18:46:47.155084 31508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:46:47.188819 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:45821
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:47.299397 31520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:47.299692 31520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:47.299770 31520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:47.303738 31520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:47.303824 31520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:47.303892 31520 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:46:47.303960 31520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:46:47.309494 31520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.159.126:45821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:47.310582 31520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:47.311699 31520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:47.321228 31528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:47.321259 31525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:47.321272 31526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:47.321417 31520 server_base.cc:1034] running on GCE node
I20260927 18:46:47.321790 31520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:47.322432 31520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:47.323583 31520 hybrid_clock.cc:648] HybridClock initialized: now 1790534807323558 us; error 35 us; skew 500 ppm
Sep 27 18:46:47 dist-test-slave-8tg2 krb5kdc[31403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534807, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:47.326257 31520 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:46:47.327075 31520 webserver.cc:460] Webserver started at http://127.29.159.65:42379/ using document root <none> and password file <none>
I20260927 18:46:47.327672 31520 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:47.327732 31520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:47.327961 31520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:47.329932 31520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "20b47dada65049329e089ff691aff47d"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "8f167a19e1b2db18dd5dc4fa2f0ad80a"
server_key_iv: "f9ed18da102e1ae50754c6bcf4f88534"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.330480 31520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "20b47dada65049329e089ff691aff47d"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "8f167a19e1b2db18dd5dc4fa2f0ad80a"
server_key_iv: "f9ed18da102e1ae50754c6bcf4f88534"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.334129 31520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 18:46:47.336748 31535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:47.338027 31520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:46:47.338217 31520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "20b47dada65049329e089ff691aff47d"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "8f167a19e1b2db18dd5dc4fa2f0ad80a"
server_key_iv: "f9ed18da102e1ae50754c6bcf4f88534"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.338336 31520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:47.350960 31520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:47.353930 31520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:47.354126 31520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:47.355682 31520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:47.355749 31520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:47.355816 31520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:47.355860 31520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:46:47 dist-test-slave-8tg2 krb5kdc[31403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534807, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:47.366452 31520 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:40365
I20260927 18:46:47.366561 31650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:40365 every 8 connection(s)
I20260927 18:46:47.367314 31520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:47.370514 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.356387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:35731 (local address 127.29.159.126:45821)
0927 18:46:47.356546 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.356553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.357921 (+ 1368us) server_negotiation.cc:407] Connection header received
0927 18:46:47.359411 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.359416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.359516 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.359611 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:47.361281 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.361965 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.363832 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.364054 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.364170 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.367013 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:47.367017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:47.367019 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:47.367050 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:47.368644 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.369123 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.369126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.369127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.369269 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.369586 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.369589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.369590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.369791 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:47.369880 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:47.370289 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 18:46:47.370393 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:46:47.371271 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.357224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:45821 (local address 127.29.159.65:35731)
0927 18:46:47.357767 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:47.357820 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:47.359172 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:47.359840 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:47.359851 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:47.360358 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:47.361106 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.361128 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.362101 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.362106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.363634 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.363648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.364206 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.364210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.364374 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.365045 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:47.365075 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:47.366862 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:47.368743 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.368747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.368776 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.369039 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.369359 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.369361 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.369363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.369484 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.369883 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:47.369888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:47.370088 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 18:46:47.370234 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 18:46:47.370465 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":133,"threads_started":1}
I20260927 18:46:47.372493 31651 heartbeater.cc:347] Connected to a master server at 127.29.159.126:45821
I20260927 18:46:47.372690 31651 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:47.373144 31651 heartbeater.cc:511] Master 127.29.159.126:45821 requested a full tablet report, sending...
I20260927 18:46:47.374606 31447 ts_manager.cc:194] Registered new tserver with Master: 20b47dada65049329e089ff691aff47d (127.29.159.65:40365)
I20260927 18:46:47.375072 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 31520
I20260927 18:46:47.375186 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:47.375525 30333 external_mini_cluster.cc:1428] Setting key a53c5033cb98f132f777eed00520f220
I20260927 18:46:47.376169 31447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:35731
I20260927 18:46:47.382848 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.374777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:54977 (local address 127.29.159.126:45821)
0927 18:46:47.374909 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.374913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.375155 (+ 242us) server_negotiation.cc:407] Connection header received
0927 18:46:47.375239 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.375242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.375287 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.375377 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:47.376037 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.376755 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.377880 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.378308 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.378431 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.379801 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:47.379804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:47.379807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:47.379831 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:47.381240 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.381671 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.381683 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.381685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.381767 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.382035 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.382039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.382041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.382215 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:47.382317 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:47.382575 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 18:46:47.382691 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:46:47.382860 31657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.374665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:45821 (local address 127.29.159.65:54977)
0927 18:46:47.375053 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:47.375067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:47.375162 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:47.375405 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:47.375408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:47.375811 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:47.375921 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.375927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.376882 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.376886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.377751 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.377762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.378602 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.378611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.378692 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.379238 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:47.379250 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:47.379667 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:47.381362 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.381369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.381372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.381574 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.381856 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.381861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.381863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.381948 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.382308 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:47.382311 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:47.382414 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 18:46:47.382523 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 18:46:47.382674 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:46:47.402278 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:45821
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:47.512681 31660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:47.512948 31660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:47.512992 31660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:47.516942 31660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:47.517001 31660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:47.517040 31660 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:46:47.517071 31660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:46:47.522214 31660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.159.126:45821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:47.523223 31660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:47.524274 31660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:47.534062 31666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:47.534047 31668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:47.534047 31665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:47.534134 31660 server_base.cc:1034] running on GCE node
I20260927 18:46:47.534667 31660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:47.535286 31660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:47.536443 31660 hybrid_clock.cc:648] HybridClock initialized: now 1790534807536396 us; error 65 us; skew 500 ppm
Sep 27 18:46:47 dist-test-slave-8tg2 krb5kdc[31403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534807, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:47.539186 31660 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:46:47.540071 31660 webserver.cc:460] Webserver started at http://127.29.159.66:40175/ using document root <none> and password file <none>
I20260927 18:46:47.540619 31660 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:47.540704 31660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:47.540927 31660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:47.543047 31660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "dd942bee40394ad1a2ca2848b9c8fb1d"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "9c8c6c909ec5730a06fdd43ff850a22a"
server_key_iv: "470b60af0dd055fff67b44ea97aaaf99"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.543649 31660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "dd942bee40394ad1a2ca2848b9c8fb1d"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "9c8c6c909ec5730a06fdd43ff850a22a"
server_key_iv: "470b60af0dd055fff67b44ea97aaaf99"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.547130 31660 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:46:47.549562 31675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:47.550709 31660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:47.550840 31660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "dd942bee40394ad1a2ca2848b9c8fb1d"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "9c8c6c909ec5730a06fdd43ff850a22a"
server_key_iv: "470b60af0dd055fff67b44ea97aaaf99"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.550944 31660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:47.580760 31660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:47.584180 31660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:47.584395 31660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:47.585803 31660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:47.585871 31660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:47.585935 31660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:47.585975 31660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:46:47 dist-test-slave-8tg2 krb5kdc[31403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534807, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:47.596922 31660 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:38275
I20260927 18:46:47.597000 31790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:38275 every 8 connection(s)
I20260927 18:46:47.598044 31660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:47.598969 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 31660
I20260927 18:46:47.599136 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:47.599534 30333 external_mini_cluster.cc:1428] Setting key b6a646bab4ef59202cd7fe15d27a8800
I20260927 18:46:47.600966 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.586997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:52181 (local address 127.29.159.126:45821)
0927 18:46:47.587168 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.587173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.588111 (+ 938us) server_negotiation.cc:407] Connection header received
0927 18:46:47.589596 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.589601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.589660 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.589739 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:47.591452 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.592219 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.593518 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.593762 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.593869 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.596807 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:47.596811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:47.596814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:47.596850 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:47.598524 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.599255 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.599259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.599261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.599361 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.599830 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.599833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.599834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.600063 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:47.600171 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:47.600667 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 18:46:47.600802 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:46:47.601761 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.587385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:45821 (local address 127.29.159.66:52181)
0927 18:46:47.587951 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:47.588016 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:47.589320 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:47.589953 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:47.589964 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:47.590488 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:47.591260 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.591283 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.592378 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.592383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.593378 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.593388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.593916 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.593920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.594102 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.594768 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:47.594791 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:47.596577 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:47.598691 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.598697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.598714 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.599117 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.599482 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.599487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.599489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.599717 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.600180 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:47.600189 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:47.600432 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 18:46:47.600610 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 18:46:47.600926 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":125,"threads_started":1}
I20260927 18:46:47.603487 31791 heartbeater.cc:347] Connected to a master server at 127.29.159.126:45821
I20260927 18:46:47.603658 31791 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:47.604125 31791 heartbeater.cc:511] Master 127.29.159.126:45821 requested a full tablet report, sending...
I20260927 18:46:47.605177 31447 ts_manager.cc:194] Registered new tserver with Master: dd942bee40394ad1a2ca2848b9c8fb1d (127.29.159.66:38275)
I20260927 18:46:47.605940 31447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:52181
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:46:47.613036 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.605791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:34495 (local address 127.29.159.126:45821)
0927 18:46:47.605930 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.605936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.605951 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:46:47.605990 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.605994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.606042 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.606170 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:47.606994 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.607548 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.608453 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.608673 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.608801 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.609878 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:47.609881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:47.609884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:47.609909 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:47.611341 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.611830 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.611838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.611840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.611912 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.612185 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.612188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.612190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.612407 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:47.612510 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:47.612774 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 18:46:47.612892 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:46:47.613049 31798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.605418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:45821 (local address 127.29.159.66:34495)
0927 18:46:47.605711 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:47.605726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:47.605810 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:47.606195 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:47.606201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:47.606723 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:47.606853 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.606861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.607664 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.607667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.608336 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.608345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.608792 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.608796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.608888 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.609408 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:47.609422 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:47.609732 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:47.611467 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.611472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.611477 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.611721 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.612001 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.612006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.612008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.612092 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.612500 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:47.612503 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:47.612595 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 18:46:47.612706 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 18:46:47.612857 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:46:47.630021 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:45821
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:47.740957 31800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:47.741205 31800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:47.741248 31800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:47.745287 31800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:47.745342 31800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:47.745380 31800 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:46:47.745419 31800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:46:47.750483 31800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.159.126:45821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:47.751416 31800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:47.752377 31800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:46:47.761914 31800 server_base.cc:1034] running on GCE node
W20260927 18:46:47.761864 31805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:47.761881 31806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:47.761852 31808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:47.762555 31800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:47.763115 31800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:47.764447 31800 hybrid_clock.cc:648] HybridClock initialized: now 1790534807764428 us; error 37 us; skew 500 ppm
Sep 27 18:46:47 dist-test-slave-8tg2 krb5kdc[31403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534807, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:47.767176 31800 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:46:47.768069 31800 webserver.cc:460] Webserver started at http://127.29.159.67:34957/ using document root <none> and password file <none>
I20260927 18:46:47.768642 31800 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:47.768729 31800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:47.768949 31800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:47.770980 31800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "9b5ac282df4649738fff29dd95854a05"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "2f28c0c654e33cca3db6f20754382406"
server_key_iv: "a0ef1e660bd2a48b233f49783d64392b"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.771597 31800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "9b5ac282df4649738fff29dd95854a05"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "2f28c0c654e33cca3db6f20754382406"
server_key_iv: "a0ef1e660bd2a48b233f49783d64392b"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.775398 31800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:46:47.777860 31815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:47.779047 31800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:47.779215 31800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "9b5ac282df4649738fff29dd95854a05"
format_stamp: "Formatted at 2026-09-27 18:46:47 on dist-test-slave-8tg2"
server_key: "2f28c0c654e33cca3db6f20754382406"
server_key_iv: "a0ef1e660bd2a48b233f49783d64392b"
server_key_version: "encryptionkey@0"
I20260927 18:46:47.779327 31800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:47.798695 31800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:47.801502 31800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:47.801698 31800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:47.803179 31800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:47.803247 31800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:47.803318 31800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:47.803363 31800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:46:47 dist-test-slave-8tg2 krb5kdc[31403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534807, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:47.814146 31800 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:37429
I20260927 18:46:47.814360 31930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:37429 every 8 connection(s)
I20260927 18:46:47.815093 31800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:47.816262 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 31800
I20260927 18:46:47.816385 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:47.816651 30333 external_mini_cluster.cc:1428] Setting key 0502eaec7ec916e0179cd82d7e120e2c
I20260927 18:46:47.817152 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.803970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:46077 (local address 127.29.159.126:45821)
0927 18:46:47.804133 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.804140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.805614 (+ 1474us) server_negotiation.cc:407] Connection header received
0927 18:46:47.807071 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.807077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.807147 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.807241 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:47.808400 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.808986 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.810114 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.810401 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.810525 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.813238 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:47.813242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:47.813245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:47.813276 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:47.814841 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.815370 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.815375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.815377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.815499 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.815896 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.815899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.815901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.816177 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:47.816298 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:47.816865 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 18:46:47.816988 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:46:47.818195 31825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.804936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:45821 (local address 127.29.159.67:46077)
0927 18:46:47.805439 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:47.805499 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:47.806820 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:47.807468 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:47.807477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:47.807802 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:47.808254 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.808267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.809106 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.809109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.809994 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.810003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.810571 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.810575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.810749 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.811363 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:47.811379 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:47.813054 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:47.814955 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.814961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.814976 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.815267 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.815626 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.815629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.815632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.815788 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.816290 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:47.816295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:47.816596 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 18:46:47.816819 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 18:46:47.817145 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":142,"threads_started":1}
I20260927 18:46:47.819391 31931 heartbeater.cc:347] Connected to a master server at 127.29.159.126:45821
I20260927 18:46:47.819552 31931 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:47.820040 31931 heartbeater.cc:511] Master 127.29.159.126:45821 requested a full tablet report, sending...
I20260927 18:46:47.821076 31448 ts_manager.cc:194] Registered new tserver with Master: 9b5ac282df4649738fff29dd95854a05 (127.29.159.67:37429)
I20260927 18:46:47.821843 31448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:46077
I20260927 18:46:47.828599 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.821553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:45821 (local address 127.29.159.67:33201)
0927 18:46:47.821865 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:47.821881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:47.821976 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:47.822250 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:47.822253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:47.822488 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:47.822584 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.822590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.823591 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.823593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.824137 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.824145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.824574 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.824578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.824645 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.825152 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:47.825165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:47.825474 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:47.827001 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.827004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.827007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.827196 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.827525 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:47.827527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:47.827529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:47.827820 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:47.828180 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:47.828182 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:47.828269 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 18:46:47.828351 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:46:47.828462 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
I20260927 18:46:47.828606 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.821653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:33201 (local address 127.29.159.126:45821)
0927 18:46:47.821841 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.821845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.821950 (+ 105us) server_negotiation.cc:407] Connection header received
0927 18:46:47.822069 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.822087 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.822141 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.822222 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:47.822694 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.823482 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.824253 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.824474 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.824562 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.825605 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:47.825608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:47.825610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:47.825632 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:47.826900 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.827293 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.827296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.827297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.827373 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.827915 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.827918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.827919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.828096 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:47.828184 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:47.828406 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 18:46:47.828498 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260927 18:46:47.830478 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:46:47 dist-test-slave-8tg2 krb5kdc[31403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:46:47 dist-test-slave-8tg2 krb5kdc[31403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:47.856844 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.848352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.29.159.126:45821)
0927 18:46:47.848511 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.848514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.848527 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:47.848568 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.848570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.848608 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.848697 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:47.849438 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.849968 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.850820 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.851042 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.851149 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.853577 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:47.853580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:47.853582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:47.853609 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:47.855144 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.855607 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.855611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.855613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.855697 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:47.855969 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:47.855974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:47.855975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:47.856166 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:47.856300 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:47.856549 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 18:46:47.856670 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:46:47.860021 31448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:47.863019 31448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:47.877244 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.871155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:40365 (local address 127.0.0.1:58284)
0927 18:46:47.871980 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:47.872033 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:47.872172 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:47.873019 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:47.873025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:47.873048 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:47.873185 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.873193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.874583 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.874586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.875714 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.875724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.876686 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.876689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.876807 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.876847 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:46:47.876961 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 18:46:47.877076 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260927 18:46:47.877247 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.871297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.29.159.65:40365)
0927 18:46:47.872440 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.872467 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.872490 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:46:47.872570 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.872576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.872822 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.873009 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:47.873299 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.874464 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.875827 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.876556 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.876681 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.876821 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:47.877003 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 18:46:47.877102 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":61,"threads_started":1}
I20260927 18:46:47.877785 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.871948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.29.159.66:38275)
0927 18:46:47.872359 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.872372 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.872478 (+ 106us) server_negotiation.cc:407] Connection header received
0927 18:46:47.872657 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.872663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.872883 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.873051 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:47.873451 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.874276 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.876147 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.876977 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.877077 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.877232 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:47.877550 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 18:46:47.877650 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":110,"threads_started":1}
I20260927 18:46:47.877708 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.871205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:38275 (local address 127.0.0.1:42744)
0927 18:46:47.872390 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:47.872403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:47.872498 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:47.873193 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:47.873196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:47.873222 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:47.873333 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.873339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.874384 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.874389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.876017 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.876027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.877298 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.877302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.877408 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.877435 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:46:47.877507 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:46:47.877583 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":219,"thread_start_us":102,"threads_started":1}
I20260927 18:46:47.878021 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.871681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.29.159.67:37429)
0927 18:46:47.872574 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.872584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.872825 (+ 241us) server_negotiation.cc:407] Connection header received
0927 18:46:47.872930 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.872937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.873104 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.873265 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:47.873552 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.874933 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.876326 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.877119 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.877213 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.877389 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:47.877778 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 18:46:47.877877 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":147,"threads_started":1}
I20260927 18:46:47.877878 31948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.872166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:37429 (local address 127.0.0.1:54154)
0927 18:46:47.872717 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:47.872730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:47.872823 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:47.873330 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:47.873333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:47.873347 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:47.873447 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.873452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.875045 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.875052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.876202 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.876208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.877459 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:47.877463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:47.877554 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:47.877576 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:46:47.877641 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:46:47.877704 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260927 18:46:47.880975 31725 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7363832aac4bec9630df7474503d96 (DEFAULT_TABLE table=test-table [id=201015a990c54beda683d6b680a0c5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:47.880985 31865 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7363832aac4bec9630df7474503d96 (DEFAULT_TABLE table=test-table [id=201015a990c54beda683d6b680a0c5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:47.880982 31585 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7363832aac4bec9630df7474503d96 (DEFAULT_TABLE table=test-table [id=201015a990c54beda683d6b680a0c5bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:47.882010 31585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7363832aac4bec9630df7474503d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:47.882010 31725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7363832aac4bec9630df7474503d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:47.882019 31865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7363832aac4bec9630df7474503d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:47.888887 31954 tablet_bootstrap.cc:492] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d: Bootstrap starting.
I20260927 18:46:47.889725 31955 tablet_bootstrap.cc:492] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d: Bootstrap starting.
I20260927 18:46:47.891050 31954 tablet_bootstrap.cc:654] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:47.891381 31955 tablet_bootstrap.cc:654] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:47.891759 31956 tablet_bootstrap.cc:492] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05: Bootstrap starting.
I20260927 18:46:47.891942 31954 log.cc:824] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:47.892151 31955 log.cc:824] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:47.893728 31956 tablet_bootstrap.cc:654] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:47.894101 31954 tablet_bootstrap.cc:492] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d: No bootstrap required, opened a new log
I20260927 18:46:47.894305 31954 ts_tablet_manager.cc:1397] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:46:47.894419 31955 tablet_bootstrap.cc:492] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d: No bootstrap required, opened a new log
I20260927 18:46:47.894447 31956 log.cc:824] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:47.894615 31955 ts_tablet_manager.cc:1397] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 18:46:47.896073 31956 tablet_bootstrap.cc:492] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05: No bootstrap required, opened a new log
I20260927 18:46:47.896251 31956 ts_tablet_manager.cc:1397] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:46:47.898303 31954 raft_consensus.cc:348] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:47.898572 31954 raft_consensus.cc:374] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:47.898631 31954 raft_consensus.cc:729] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd942bee40394ad1a2ca2848b9c8fb1d, State: Initialized, Role: FOLLOWER
I20260927 18:46:47.898634 31955 raft_consensus.cc:348] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:47.898886 31955 raft_consensus.cc:374] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:47.898943 31955 raft_consensus.cc:729] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b47dada65049329e089ff691aff47d, State: Initialized, Role: FOLLOWER
I20260927 18:46:47.899130 31954 consensus_queue.cc:260] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:47.899088 31956 raft_consensus.cc:348] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:47.899350 31956 raft_consensus.cc:374] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:47.899405 31956 raft_consensus.cc:729] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5ac282df4649738fff29dd95854a05, State: Initialized, Role: FOLLOWER
I20260927 18:46:47.899588 31955 consensus_queue.cc:260] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:47.899874 31956 consensus_queue.cc:260] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:47.900189 31954 ts_tablet_manager.cc:1428] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 18:46:47.900461 31955 ts_tablet_manager.cc:1428] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:46:47.900539 31931 heartbeater.cc:503] Master 127.29.159.126:45821 was elected leader, sending a full tablet report...
I20260927 18:46:47.900549 31791 heartbeater.cc:503] Master 127.29.159.126:45821 was elected leader, sending a full tablet report...
I20260927 18:46:47.900825 31956 ts_tablet_manager.cc:1428] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:46:47.901481 31651 heartbeater.cc:503] Master 127.29.159.126:45821 was elected leader, sending a full tablet report...
I20260927 18:46:47.996680 31960 raft_consensus.cc:482] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:47.996889 31960 raft_consensus.cc:504] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:47.998021 31960 leader_election.cc:288] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20b47dada65049329e089ff691aff47d (127.29.159.65:40365), 9b5ac282df4649738fff29dd95854a05 (127.29.159.67:37429)
I20260927 18:46:48.002149 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.998433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:35591 (local address 127.29.159.67:37429)
0927 18:46:47.998578 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.998583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.998599 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:46:47.998776 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.998779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.998850 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.998927 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:47.999241 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.999883 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.000988 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.001585 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.001710 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.001762 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.001991 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.002046 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:46:48.002170 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.998337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:37429 (local address 127.29.159.66:35591)
0927 18:46:47.998489 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:47.998523 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:47.998635 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:47.998963 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:47.998969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:47.998997 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:47.999125 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.999136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:47.999996 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.000000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.000866 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.000876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.001706 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.001709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.001864 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.001890 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:46:48.001972 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:46:48.002039 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 18:46:48.002864 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.998337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:40365 (local address 127.29.159.66:58921)
0927 18:46:47.998801 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:47.998814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:47.998915 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:47.999174 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:47.999176 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:47.999209 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:47.999334 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:47.999340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.000182 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.000184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.001566 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.001576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.002455 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.002459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.002559 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.002582 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:46:48.002660 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:46:48.002729 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":119,"thread_start_us":128,"threads_started":1}
I20260927 18:46:48.002871 31885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7363832aac4bec9630df7474503d96" candidate_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5ac282df4649738fff29dd95854a05" is_pre_election: true
I20260927 18:46:48.002964 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:47.998606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:58921 (local address 127.29.159.65:40365)
0927 18:46:47.998738 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:47.998744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:47.998891 (+ 147us) server_negotiation.cc:407] Connection header received
0927 18:46:47.999026 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:47.999029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:47.999076 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:47.999222 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:47.999514 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.000068 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.001700 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.002305 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.002432 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.002499 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.002687 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.002754 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:46:48.003194 31885 raft_consensus.cc:2454] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd942bee40394ad1a2ca2848b9c8fb1d in term 0.
I20260927 18:46:48.003611 31605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7363832aac4bec9630df7474503d96" candidate_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b47dada65049329e089ff691aff47d" is_pre_election: true
I20260927 18:46:48.003840 31679 leader_election.cc:302] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5ac282df4649738fff29dd95854a05, dd942bee40394ad1a2ca2848b9c8fb1d; no voters:
I20260927 18:46:48.003968 31605 raft_consensus.cc:2454] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd942bee40394ad1a2ca2848b9c8fb1d in term 0.
I20260927 18:46:48.004130 31960 raft_consensus.cc:2781] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:48.004222 31960 raft_consensus.cc:482] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:48.004297 31960 raft_consensus.cc:3037] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:48.005581 31960 raft_consensus.cc:504] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.005973 31960 leader_election.cc:288] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [CANDIDATE]: Term 1 election: Requested vote from peers 20b47dada65049329e089ff691aff47d (127.29.159.65:40365), 9b5ac282df4649738fff29dd95854a05 (127.29.159.67:37429)
I20260927 18:46:48.006291 31885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7363832aac4bec9630df7474503d96" candidate_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5ac282df4649738fff29dd95854a05"
I20260927 18:46:48.006283 31605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7363832aac4bec9630df7474503d96" candidate_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b47dada65049329e089ff691aff47d"
I20260927 18:46:48.006417 31605 raft_consensus.cc:3037] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:48.006417 31885 raft_consensus.cc:3037] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:48.007824 31605 raft_consensus.cc:2454] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd942bee40394ad1a2ca2848b9c8fb1d in term 1.
I20260927 18:46:48.008153 31885 raft_consensus.cc:2454] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd942bee40394ad1a2ca2848b9c8fb1d in term 1.
I20260927 18:46:48.008155 31677 leader_election.cc:302] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b47dada65049329e089ff691aff47d, dd942bee40394ad1a2ca2848b9c8fb1d; no voters:
I20260927 18:46:48.008342 31960 raft_consensus.cc:2781] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:48.008585 31960 raft_consensus.cc:686] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 1 LEADER]: Becoming Leader. State: Replica: dd942bee40394ad1a2ca2848b9c8fb1d, State: Running, Role: LEADER
I20260927 18:46:48.008966 31960 consensus_queue.cc:237] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.012252 31448 catalog_manager.cc:5495] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d reported cstate change: term changed from 0 to 1, leader changed from <none> to dd942bee40394ad1a2ca2848b9c8fb1d (127.29.159.66). New cstate: current_term: 1 leader_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } health_report { overall_health: UNKNOWN } } }
I20260927 18:46:48.032737 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.029477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.29.159.66:38275)
0927 18:46:48.029590 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.029593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.029608 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:46:48.029798 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.029800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.029853 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.029936 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:48.030141 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.030652 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.031519 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.031739 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.031818 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.032064 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:48.032444 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:48.032525 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.032591 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.032635 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:46:48.038355 31885 raft_consensus.cc:1261] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05 [term 1 FOLLOWER]: Refusing update from remote peer dd942bee40394ad1a2ca2848b9c8fb1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:48.038491 31605 raft_consensus.cc:1261] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d [term 1 FOLLOWER]: Refusing update from remote peer dd942bee40394ad1a2ca2848b9c8fb1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:48.039069 31960 consensus_queue.cc:1035] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:48.039238 31964 consensus_queue.cc:1035] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:48.048079 31967 mvcc.cc:205] Tried to move back new op lower bound from 7334030573717594112 to 7334030573607522304. Current Snapshot: MvccSnapshot[applied={T|T < 7334030573717594112 or (T in {7334030573717594112})}]
I20260927 18:46:48.057560 31448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:48.057739 31448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:48.060099 31448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97: Sending DeleteTablet for 3 replicas of tablet 7d7363832aac4bec9630df7474503d96
I20260927 18:46:48.060849 31725 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7363832aac4bec9630df7474503d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:48 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:42744
I20260927 18:46:48.060901 31585 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7363832aac4bec9630df7474503d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:48 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:58284
I20260927 18:46:48.060940 31865 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7363832aac4bec9630df7474503d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:48 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:54154
I20260927 18:46:48.061352 31978 tablet_replica.cc:330] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05: stopping tablet replica
I20260927 18:46:48.061477 31977 tablet_replica.cc:330] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d: stopping tablet replica
I20260927 18:46:48.061661 31978 raft_consensus.cc:2229] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:48.061713 31977 raft_consensus.cc:2229] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:46:48.061960 31978 raft_consensus.cc:2258] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:48.062103 31448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:48.062497 31448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:48.062198 31977 raft_consensus.cc:2258] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:48.063254 31979 tablet_replica.cc:330] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d: stopping tablet replica
I20260927 18:46:48.063570 31979 raft_consensus.cc:2229] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:48.063846 31979 raft_consensus.cc:2258] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:48.064494 31977 ts_tablet_manager.cc:1905] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:48.065122 31979 ts_tablet_manager.cc:1905] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:48.067453 31977 ts_tablet_manager.cc:1918] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:48.067611 31977 log.cc:1196] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/7d7363832aac4bec9630df7474503d96
I20260927 18:46:48.067791 31979 ts_tablet_manager.cc:1918] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:48.067888 31979 log.cc:1196] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/wal/wals/7d7363832aac4bec9630df7474503d96
I20260927 18:46:48.068149 31977 ts_tablet_manager.cc:1939] T 7d7363832aac4bec9630df7474503d96 P dd942bee40394ad1a2ca2848b9c8fb1d: Deleting consensus metadata
I20260927 18:46:48.068198 31979 ts_tablet_manager.cc:1939] T 7d7363832aac4bec9630df7474503d96 P 20b47dada65049329e089ff691aff47d: Deleting consensus metadata
I20260927 18:46:48.069589 31434 catalog_manager.cc:4841] TS dd942bee40394ad1a2ca2848b9c8fb1d (127.29.159.66:38275): tablet 7d7363832aac4bec9630df7474503d96 (table test-table [id=201015a990c54beda683d6b680a0c5bd]) successfully deleted
I20260927 18:46:48.069641 31725 tablet_service.cc:1467] Processing CreateTablet for tablet 718e598388784bde9ff33147bc2fc910 (DEFAULT_TABLE table=test-table [id=ac3fb7acf56c4634a394281a9109d042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:48.069881 31725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718e598388784bde9ff33147bc2fc910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:48.069835 31865 tablet_service.cc:1467] Processing CreateTablet for tablet 718e598388784bde9ff33147bc2fc910 (DEFAULT_TABLE table=test-table [id=ac3fb7acf56c4634a394281a9109d042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:48.070138 31865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718e598388784bde9ff33147bc2fc910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:48.071750 31433 catalog_manager.cc:4841] TS 20b47dada65049329e089ff691aff47d (127.29.159.65:40365): tablet 7d7363832aac4bec9630df7474503d96 (table test-table [id=201015a990c54beda683d6b680a0c5bd]) successfully deleted
I20260927 18:46:48.071991 31954 tablet_bootstrap.cc:492] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d: Bootstrap starting.
I20260927 18:46:48.072371 31956 tablet_bootstrap.cc:492] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05: Bootstrap starting.
I20260927 18:46:48.072347 31585 tablet_service.cc:1467] Processing CreateTablet for tablet 718e598388784bde9ff33147bc2fc910 (DEFAULT_TABLE table=test-table [id=ac3fb7acf56c4634a394281a9109d042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:48.072599 31585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718e598388784bde9ff33147bc2fc910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:48.072818 31978 ts_tablet_manager.cc:1905] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:48.072980 31954 tablet_bootstrap.cc:654] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:48.073323 31956 tablet_bootstrap.cc:654] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:48.074877 31955 tablet_bootstrap.cc:492] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d: Bootstrap starting.
I20260927 18:46:48.075117 31956 tablet_bootstrap.cc:492] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05: No bootstrap required, opened a new log
I20260927 18:46:48.075207 31956 ts_tablet_manager.cc:1397] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:46:48.075693 31978 ts_tablet_manager.cc:1918] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:46:48.075729 31956 raft_consensus.cc:348] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.075803 31955 tablet_bootstrap.cc:654] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:48.075847 31956 raft_consensus.cc:374] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:48.075886 31956 raft_consensus.cc:729] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5ac282df4649738fff29dd95854a05, State: Initialized, Role: FOLLOWER
I20260927 18:46:48.076014 31978 log.cc:1196] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/7d7363832aac4bec9630df7474503d96
I20260927 18:46:48.076027 31956 consensus_queue.cc:260] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.076341 31956 ts_tablet_manager.cc:1428] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:46:48.076476 31978 ts_tablet_manager.cc:1939] T 7d7363832aac4bec9630df7474503d96 P 9b5ac282df4649738fff29dd95854a05: Deleting consensus metadata
I20260927 18:46:48.077664 31955 tablet_bootstrap.cc:492] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d: No bootstrap required, opened a new log
I20260927 18:46:48.077740 31955 ts_tablet_manager.cc:1397] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 18:46:48.078300 31954 tablet_bootstrap.cc:492] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d: No bootstrap required, opened a new log
I20260927 18:46:48.078397 31954 ts_tablet_manager.cc:1397] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 18:46:48.078388 31955 raft_consensus.cc:348] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.078504 31955 raft_consensus.cc:374] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:48.078550 31955 raft_consensus.cc:729] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b47dada65049329e089ff691aff47d, State: Initialized, Role: FOLLOWER
I20260927 18:46:48.078671 31955 consensus_queue.cc:260] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.078945 31955 ts_tablet_manager.cc:1428] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260927 18:46:48.078883 31954 raft_consensus.cc:348] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.078920 31435 catalog_manager.cc:4841] TS 9b5ac282df4649738fff29dd95854a05 (127.29.159.67:37429): tablet 7d7363832aac4bec9630df7474503d96 (table test-table [id=201015a990c54beda683d6b680a0c5bd]) successfully deleted
I20260927 18:46:48.078995 31954 raft_consensus.cc:374] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:48.079052 31954 raft_consensus.cc:729] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd942bee40394ad1a2ca2848b9c8fb1d, State: Initialized, Role: FOLLOWER
I20260927 18:46:48.079186 31954 consensus_queue.cc:260] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.079495 31954 ts_tablet_manager.cc:1428] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20260927 18:46:48.100332 31792 tablet.cc:2369] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:46:48.118878 31652 tablet.cc:2369] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:48.230587 31961 raft_consensus.cc:482] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:48.230741 31961 raft_consensus.cc:504] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.231962 31961 leader_election.cc:288] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd942bee40394ad1a2ca2848b9c8fb1d (127.29.159.66:38275), 9b5ac282df4649738fff29dd95854a05 (127.29.159.67:37429)
I20260927 18:46:48.236040 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.232345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:53605 (local address 127.29.159.67:37429)
0927 18:46:48.232620 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.232626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.232668 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:46:48.232862 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.232865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.232913 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.232989 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:48.233393 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.233901 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.234968 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.235399 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.235562 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.235623 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.235846 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.235918 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260927 18:46:48.236069 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.232260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:37429 (local address 127.29.159.65:53605)
0927 18:46:48.232577 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:48.232593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:48.232730 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:48.233014 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:48.233017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:48.233033 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:48.233280 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.233285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.234006 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.234012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.234846 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.234855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.235575 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.235582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.235724 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.235748 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:46:48.235828 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 18:46:48.235901 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":43,"thread_start_us":106,"threads_started":1}
I20260927 18:46:48.236136 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.232385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:54627 (local address 127.29.159.66:38275)
0927 18:46:48.232535 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.232548 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.232749 (+ 201us) server_negotiation.cc:407] Connection header received
0927 18:46:48.232914 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.232917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.232958 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.233036 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:48.233446 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.233974 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.235032 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.235549 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.235683 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.235742 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.235939 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.236010 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:46:48.236246 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.232270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:38275 (local address 127.29.159.65:54627)
0927 18:46:48.232649 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:48.232661 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:48.232798 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:48.233058 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:48.233061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:48.233077 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:48.233328 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.233334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.234082 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.234088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.234902 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.234911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.235681 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.235684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.235805 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.235834 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:46:48.235920 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:46:48.235988 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":119,"thread_start_us":109,"threads_started":1}
I20260927 18:46:48.236670 31885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e598388784bde9ff33147bc2fc910" candidate_uuid: "20b47dada65049329e089ff691aff47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5ac282df4649738fff29dd95854a05" is_pre_election: true
I20260927 18:46:48.236862 31885 raft_consensus.cc:2454] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b47dada65049329e089ff691aff47d in term 0.
I20260927 18:46:48.236878 31745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e598388784bde9ff33147bc2fc910" candidate_uuid: "20b47dada65049329e089ff691aff47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" is_pre_election: true
I20260927 18:46:48.237145 31745 raft_consensus.cc:2454] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b47dada65049329e089ff691aff47d in term 0.
I20260927 18:46:48.237273 31539 leader_election.cc:302] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b47dada65049329e089ff691aff47d, 9b5ac282df4649738fff29dd95854a05; no voters:
I20260927 18:46:48.237557 31961 raft_consensus.cc:2781] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:48.237620 31961 raft_consensus.cc:482] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:48.237648 31961 raft_consensus.cc:3037] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:48.238655 31961 raft_consensus.cc:504] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.239027 31961 leader_election.cc:288] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [CANDIDATE]: Term 1 election: Requested vote from peers dd942bee40394ad1a2ca2848b9c8fb1d (127.29.159.66:38275), 9b5ac282df4649738fff29dd95854a05 (127.29.159.67:37429)
I20260927 18:46:48.239377 31885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e598388784bde9ff33147bc2fc910" candidate_uuid: "20b47dada65049329e089ff691aff47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5ac282df4649738fff29dd95854a05"
I20260927 18:46:48.239451 31745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e598388784bde9ff33147bc2fc910" candidate_uuid: "20b47dada65049329e089ff691aff47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d"
I20260927 18:46:48.239557 31885 raft_consensus.cc:3037] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:48.239566 31745 raft_consensus.cc:3037] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:48.240490 31745 raft_consensus.cc:2454] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b47dada65049329e089ff691aff47d in term 1.
I20260927 18:46:48.240511 31885 raft_consensus.cc:2454] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b47dada65049329e089ff691aff47d in term 1.
I20260927 18:46:48.240775 31538 leader_election.cc:302] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b47dada65049329e089ff691aff47d, dd942bee40394ad1a2ca2848b9c8fb1d; no voters:
I20260927 18:46:48.240969 31961 raft_consensus.cc:2781] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:48.241267 31961 raft_consensus.cc:686] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 1 LEADER]: Becoming Leader. State: Replica: 20b47dada65049329e089ff691aff47d, State: Running, Role: LEADER
I20260927 18:46:48.241528 31961 consensus_queue.cc:237] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.243944 31448 catalog_manager.cc:5495] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b47dada65049329e089ff691aff47d (127.29.159.65). New cstate: current_term: 1 leader_uuid: "20b47dada65049329e089ff691aff47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:46:48 dist-test-slave-8tg2 krb5kdc[31403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534807, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:48.285329 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.277264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:45821 (local address 127.0.0.1:46292)
0927 18:46:48.277607 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:48.277625 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:48.277745 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:48.277989 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:48.277991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:48.278405 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:48.278496 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.278501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.279205 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.279207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.279777 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.279784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.280185 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.280187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.280247 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.280841 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:48.280853 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:48.282249 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:48.283969 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:48.283977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:48.283982 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:48.284204 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:48.284437 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:48.284439 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:48.284441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:48.284514 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:48.284854 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:48.284860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:48.284961 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 18:46:48.285051 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 18:46:48.285167 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
I20260927 18:46:48.285346 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.277396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.29.159.126:45821)
0927 18:46:48.277525 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.277529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.277714 (+ 185us) server_negotiation.cc:407] Connection header received
0927 18:46:48.277831 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.277833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.277874 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.277955 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:48.278607 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.279115 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.279884 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.280092 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.280172 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.282374 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:48.282377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:48.282379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:48.282399 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:48.283867 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:48.284288 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:48.284290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:48.284291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:48.284363 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:48.284583 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:48.284585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:48.284586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:48.284771 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:48.284847 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.285101 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.285220 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":45}
I20260927 18:46:48.289153 31448 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:46292:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:46:48.289897 31448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:48.294531 31585 tablet_service.cc:1467] Processing CreateTablet for tablet bbd32158342946b68e85232a61954fde (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be8569fdc1e84a938da82707d926570f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:46:48.294544 31725 tablet_service.cc:1467] Processing CreateTablet for tablet bbd32158342946b68e85232a61954fde (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be8569fdc1e84a938da82707d926570f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:46:48.294552 31865 tablet_service.cc:1467] Processing CreateTablet for tablet bbd32158342946b68e85232a61954fde (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be8569fdc1e84a938da82707d926570f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:46:48.294924 31585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd32158342946b68e85232a61954fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:48.294919 31865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd32158342946b68e85232a61954fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:48.294919 31725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd32158342946b68e85232a61954fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:48.297324 31955 tablet_bootstrap.cc:492] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d: Bootstrap starting.
I20260927 18:46:48.297680 31954 tablet_bootstrap.cc:492] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d: Bootstrap starting.
I20260927 18:46:48.297740 31956 tablet_bootstrap.cc:492] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: Bootstrap starting.
I20260927 18:46:48.298535 31954 tablet_bootstrap.cc:654] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:48.298535 31955 tablet_bootstrap.cc:654] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:48.298794 31956 tablet_bootstrap.cc:654] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:48.300207 31956 tablet_bootstrap.cc:492] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: No bootstrap required, opened a new log
I20260927 18:46:48.300302 31956 ts_tablet_manager.cc:1397] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:46:48.300590 31954 tablet_bootstrap.cc:492] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d: No bootstrap required, opened a new log
I20260927 18:46:48.300638 31955 tablet_bootstrap.cc:492] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d: No bootstrap required, opened a new log
I20260927 18:46:48.300669 31954 ts_tablet_manager.cc:1397] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 18:46:48.300704 31955 ts_tablet_manager.cc:1397] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 18:46:48.300730 31956 raft_consensus.cc:348] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.300858 31956 raft_consensus.cc:374] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:48.300900 31956 raft_consensus.cc:729] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5ac282df4649738fff29dd95854a05, State: Initialized, Role: FOLLOWER
I20260927 18:46:48.301029 31956 consensus_queue.cc:260] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.301059 31954 raft_consensus.cc:348] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.301172 31954 raft_consensus.cc:374] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:48.301119 31955 raft_consensus.cc:348] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.301203 31954 raft_consensus.cc:729] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd942bee40394ad1a2ca2848b9c8fb1d, State: Initialized, Role: FOLLOWER
I20260927 18:46:48.301218 31955 raft_consensus.cc:374] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:48.301254 31955 raft_consensus.cc:729] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b47dada65049329e089ff691aff47d, State: Initialized, Role: FOLLOWER
I20260927 18:46:48.301342 31954 consensus_queue.cc:260] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.301362 31955 consensus_queue.cc:260] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.301334 31962 tablet_replica.cc:439] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } } }
I20260927 18:46:48.301507 31962 tablet_replica.cc:442] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:48.301604 31954 ts_tablet_manager.cc:1428] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:46:48.301604 31955 ts_tablet_manager.cc:1428] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 18:46:48.301604 31961 tablet_replica.cc:439] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } } }
I20260927 18:46:48.302085 31961 tablet_replica.cc:442] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:48.302039 31974 tablet_replica.cc:439] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } } }
I20260927 18:46:48.302163 31974 tablet_replica.cc:442] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:48.302407 31956 ts_tablet_manager.cc:1428] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260927 18:46:48.316214 31932 tablet.cc:2369] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:48.403756 31962 raft_consensus.cc:482] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:48.403942 31962 raft_consensus.cc:504] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.405189 31962 leader_election.cc:288] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20b47dada65049329e089ff691aff47d (127.29.159.65:40365), dd942bee40394ad1a2ca2848b9c8fb1d (127.29.159.66:38275)
I20260927 18:46:48.410591 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.405556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:60453 (local address 127.29.159.66:38275)
0927 18:46:48.405694 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.405699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.405943 (+ 244us) server_negotiation.cc:407] Connection header received
0927 18:46:48.406075 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.406078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.406129 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.406217 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:48.406635 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.407440 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.409282 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.409882 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.410009 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.410063 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.410405 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.410471 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:46:48.410591 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.405569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:50799 (local address 127.29.159.65:40365)
0927 18:46:48.405694 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.405699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.405902 (+ 203us) server_negotiation.cc:407] Connection header received
0927 18:46:48.406076 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.406079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.406129 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.406218 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:48.406631 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.407409 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.409286 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.409896 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.410009 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.410063 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.410390 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.410462 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:46:48.410665 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.405438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:38275 (local address 127.29.159.67:60453)
0927 18:46:48.405862 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:48.405875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:48.405970 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:48.406239 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:48.406242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:48.406266 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:48.406521 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.406529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.407543 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.407547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.409147 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.409159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.410060 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.410065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.410280 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.410315 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:46:48.410391 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 18:46:48.410452 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":321,"spinlock_wait_cycles":17280,"thread_start_us":73,"threads_started":1}
I20260927 18:46:48.410683 31996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.405438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:40365 (local address 127.29.159.67:50799)
0927 18:46:48.405817 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:48.405834 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:48.405970 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:48.406239 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:48.406242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:48.406266 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:48.406528 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.406534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.407543 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.407547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.409150 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.409159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.410060 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.410065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.410222 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.410259 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:46:48.410361 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 18:46:48.410452 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":236,"spinlock_wait_cycles":27008,"thread_start_us":88,"threads_started":1}
I20260927 18:46:48.411407 31745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd32158342946b68e85232a61954fde" candidate_uuid: "9b5ac282df4649738fff29dd95854a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" is_pre_election: true
I20260927 18:46:48.411455 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd32158342946b68e85232a61954fde" candidate_uuid: "9b5ac282df4649738fff29dd95854a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b47dada65049329e089ff691aff47d" is_pre_election: true
I20260927 18:46:48.411597 31745 raft_consensus.cc:2454] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b5ac282df4649738fff29dd95854a05 in term 0.
I20260927 18:46:48.411597 31604 raft_consensus.cc:2454] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b5ac282df4649738fff29dd95854a05 in term 0.
I20260927 18:46:48.411979 31817 leader_election.cc:302] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b47dada65049329e089ff691aff47d, 9b5ac282df4649738fff29dd95854a05; no voters:
I20260927 18:46:48.412298 31962 raft_consensus.cc:2781] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:48.412369 31962 raft_consensus.cc:482] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:48.412423 31962 raft_consensus.cc:3037] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:48.413452 31962 raft_consensus.cc:504] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.413798 31962 leader_election.cc:288] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [CANDIDATE]: Term 1 election: Requested vote from peers 20b47dada65049329e089ff691aff47d (127.29.159.65:40365), dd942bee40394ad1a2ca2848b9c8fb1d (127.29.159.66:38275)
I20260927 18:46:48.414188 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd32158342946b68e85232a61954fde" candidate_uuid: "9b5ac282df4649738fff29dd95854a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b47dada65049329e089ff691aff47d"
I20260927 18:46:48.414330 31604 raft_consensus.cc:3037] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:48.414295 31745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd32158342946b68e85232a61954fde" candidate_uuid: "9b5ac282df4649738fff29dd95854a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d"
I20260927 18:46:48.414425 31745 raft_consensus.cc:3037] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:48.415650 31745 raft_consensus.cc:2454] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5ac282df4649738fff29dd95854a05 in term 1.
I20260927 18:46:48.415650 31604 raft_consensus.cc:2454] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5ac282df4649738fff29dd95854a05 in term 1.
I20260927 18:46:48.415923 31817 leader_election.cc:302] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b47dada65049329e089ff691aff47d, 9b5ac282df4649738fff29dd95854a05; no voters:
I20260927 18:46:48.416126 31962 raft_consensus.cc:2781] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:48.416323 31962 raft_consensus.cc:686] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [term 1 LEADER]: Becoming Leader. State: Replica: 9b5ac282df4649738fff29dd95854a05, State: Running, Role: LEADER
I20260927 18:46:48.416618 31962 consensus_queue.cc:237] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } }
I20260927 18:46:48.417893 31998 tablet_replica.cc:439] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: TxnStatusTablet state changed. Reason: New leader 9b5ac282df4649738fff29dd95854a05. Latest consensus state: current_term: 1 leader_uuid: "9b5ac282df4649738fff29dd95854a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } } }
I20260927 18:46:48.418011 31998 tablet_replica.cc:442] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:46:48.418319 32000 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:46:48.418476 32000 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:46:48.419296 31448 catalog_manager.cc:5495] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b5ac282df4649738fff29dd95854a05 (127.29.159.67). New cstate: current_term: 1 leader_uuid: "9b5ac282df4649738fff29dd95854a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } health_report { overall_health: HEALTHY } } }
I20260927 18:46:48.433024 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.428845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:37429 (local address 127.0.0.1:54164)
0927 18:46:48.429120 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:48.429133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:48.429230 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:48.429550 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:48.429552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:48.429557 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:46:48.429648 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.429651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.430634 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.430637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.431283 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.431292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.431910 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.431913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.431977 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.432020 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:46:48.432771 (+ 751us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:46:48.432787 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 18:46:48.432849 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:46:48.432917 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260927 18:46:48.433061 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.428975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.29.159.67:37429)
0927 18:46:48.429102 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.429106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.429188 (+ 82us) server_negotiation.cc:407] Connection header received
0927 18:46:48.429359 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.429362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.429436 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.429547 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:48.429742 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.430531 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.431396 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.431805 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.431914 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.432154 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:48.432670 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:48.432779 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.432863 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.432916 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:46:48.435940 31604 raft_consensus.cc:1261] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d [term 1 FOLLOWER]: Refusing update from remote peer 9b5ac282df4649738fff29dd95854a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:48.436123 31745 raft_consensus.cc:1261] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d [term 1 FOLLOWER]: Refusing update from remote peer 9b5ac282df4649738fff29dd95854a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:48.436477 31998 consensus_queue.cc:1035] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:48.436694 31962 consensus_queue.cc:1035] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:48.438241 31974 tablet_replica.cc:439] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d: TxnStatusTablet state changed. Reason: New leader 9b5ac282df4649738fff29dd95854a05. Latest consensus state: current_term: 1 leader_uuid: "9b5ac282df4649738fff29dd95854a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } } }
I20260927 18:46:48.438367 31974 tablet_replica.cc:442] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:48.439706 31962 tablet_replica.cc:439] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b5ac282df4649738fff29dd95854a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } } }
I20260927 18:46:48.439818 31962 tablet_replica.cc:442] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:46:48.440694 31985 tablet_replica.cc:439] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b5ac282df4649738fff29dd95854a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } } }
I20260927 18:46:48.440822 31985 tablet_replica.cc:442] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:48.441855 31969 mvcc.cc:205] Tried to move back new op lower bound from 7334030575348580352 to 7334030575277027328. Current Snapshot: MvccSnapshot[applied={T|T < 7334030575348580352 or (T in {7334030575348580352})}]
I20260927 18:46:48.442945 31961 tablet_replica.cc:439] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d: TxnStatusTablet state changed. Reason: New leader 9b5ac282df4649738fff29dd95854a05. Latest consensus state: current_term: 1 leader_uuid: "9b5ac282df4649738fff29dd95854a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } } }
I20260927 18:46:48.443073 31961 tablet_replica.cc:442] T bbd32158342946b68e85232a61954fde P 20b47dada65049329e089ff691aff47d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:48.444763 31998 tablet_replica.cc:439] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b5ac282df4649738fff29dd95854a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } } }
I20260927 18:46:48.444875 31998 tablet_replica.cc:442] T bbd32158342946b68e85232a61954fde P 9b5ac282df4649738fff29dd95854a05: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:46:48.445393 31974 tablet_replica.cc:439] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b5ac282df4649738fff29dd95854a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b47dada65049329e089ff691aff47d" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 40365 } } peers { permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 } } peers { permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 } } }
I20260927 18:46:48.445508 31974 tablet_replica.cc:442] T bbd32158342946b68e85232a61954fde P dd942bee40394ad1a2ca2848b9c8fb1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:48.450737 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.446987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58288 (local address 127.29.159.65:40365)
0927 18:46:48.447089 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.447094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.447105 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:46:48.447170 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.447173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.447234 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.447355 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:48.447602 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.448211 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.449392 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.449620 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.449709 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.449976 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:48.450375 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:48.450459 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.450547 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.450596 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:46:48.461664 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.457833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:37429 (local address 127.29.159.65:33865)
0927 18:46:48.458270 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:48.458283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:48.458370 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:48.458606 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:48.458608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:48.458614 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:46:48.458714 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.458718 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.459731 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.459734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.460364 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.460372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.460973 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.460975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.461048 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.461089 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:46:48.461395 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:46:48.461401 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:46:48.461466 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:46:48.461528 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":115,"threads_started":1}
I20260927 18:46:48.461674 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.457968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:33865 (local address 127.29.159.67:37429)
0927 18:46:48.458082 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.458086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.458348 (+ 262us) server_negotiation.cc:407] Connection header received
0927 18:46:48.458455 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.458459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.458509 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.458582 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:48.458806 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.459622 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.460475 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.460861 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.460959 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.461206 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:48.461285 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:48.461379 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.461489 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.461542 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:46:48.468286 31745 raft_consensus.cc:1261] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d [term 1 FOLLOWER]: Refusing update from remote peer 20b47dada65049329e089ff691aff47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:48.468484 31885 raft_consensus.cc:1261] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05 [term 1 FOLLOWER]: Refusing update from remote peer 20b47dada65049329e089ff691aff47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:48.468796 31985 consensus_queue.cc:1035] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd942bee40394ad1a2ca2848b9c8fb1d" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 38275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:48.468957 31961 consensus_queue.cc:1035] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5ac282df4649738fff29dd95854a05" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:48.474097 31968 mvcc.cc:205] Tried to move back new op lower bound from 7334030575481806848 to 7334030574559809536. Current Snapshot: MvccSnapshot[applied={T|T < 7334030575481806848}]
I20260927 18:46:48.492352 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.488508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:40365 (local address 127.29.159.67:47379)
0927 18:46:48.488848 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:48.488864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:48.488965 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:48.489189 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:48.489192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:48.489200 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:46:48.489350 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.489357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.490142 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.490148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.491069 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:48.491091 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.491723 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.491726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:48.491799 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.491842 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:46:48.492118 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:46:48.492122 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:46:48.492164 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 18:46:48.492224 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
I20260927 18:46:48.492348 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.488860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:47379 (local address 127.29.159.65:40365)
0927 18:46:48.488982 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.488987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.489004 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:46:48.489051 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.489054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.489096 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.489178 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:48.489461 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.490050 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.491239 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.491611 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.491712 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.491961 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:48.492044 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:48.492118 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.492173 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.492221 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:46:48.519721 31448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:48.519883 31448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:48.522053 31448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3a37401665e49028f25192242ce5a97: Sending DeleteTablet for 3 replicas of tablet 718e598388784bde9ff33147bc2fc910
I20260927 18:46:48.522699 31585 tablet_service.cc:1514] Processing DeleteTablet for tablet 718e598388784bde9ff33147bc2fc910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:48 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:58284
I20260927 18:46:48.522935 31865 tablet_service.cc:1514] Processing DeleteTablet for tablet 718e598388784bde9ff33147bc2fc910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:48 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:54154
I20260927 18:46:48.523043 31979 tablet_replica.cc:330] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d: stopping tablet replica
I20260927 18:46:48.523114 31978 tablet_replica.cc:330] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05: stopping tablet replica
I20260927 18:46:48.523151 31979 raft_consensus.cc:2229] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:46:48.523247 31978 raft_consensus.cc:2229] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:48.523356 31978 raft_consensus.cc:2258] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:48.523458 31979 raft_consensus.cc:2258] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:48.523626 31725 tablet_service.cc:1514] Processing DeleteTablet for tablet 718e598388784bde9ff33147bc2fc910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:48 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:42744
I20260927 18:46:48.523880 31977 tablet_replica.cc:330] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d: stopping tablet replica
I20260927 18:46:48.523993 31977 raft_consensus.cc:2229] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:48.524093 31977 raft_consensus.cc:2258] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:48.524467 31978 ts_tablet_manager.cc:1905] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:48.524839 31977 ts_tablet_manager.cc:1905] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:48.527455 31978 ts_tablet_manager.cc:1918] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:46:48.527530 31978 log.cc:1196] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/718e598388784bde9ff33147bc2fc910
I20260927 18:46:48.527799 31978 ts_tablet_manager.cc:1939] T 718e598388784bde9ff33147bc2fc910 P 9b5ac282df4649738fff29dd95854a05: Deleting consensus metadata
I20260927 18:46:48.528031 31979 ts_tablet_manager.cc:1905] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:48.530786 31979 ts_tablet_manager.cc:1918] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:46:48.530867 31979 log.cc:1196] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-0/wal/wals/718e598388784bde9ff33147bc2fc910
I20260927 18:46:48.531117 31979 ts_tablet_manager.cc:1939] T 718e598388784bde9ff33147bc2fc910 P 20b47dada65049329e089ff691aff47d: Deleting consensus metadata
I20260927 18:46:48.531569 31435 catalog_manager.cc:4841] TS 9b5ac282df4649738fff29dd95854a05 (127.29.159.67:37429): tablet 718e598388784bde9ff33147bc2fc910 (table test-table [id=ac3fb7acf56c4634a394281a9109d042]) successfully deleted
I20260927 18:46:48.532003 31977 ts_tablet_manager.cc:1918] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:46:48.532083 31977 log.cc:1196] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/718e598388784bde9ff33147bc2fc910
I20260927 18:46:48.532333 31977 ts_tablet_manager.cc:1939] T 718e598388784bde9ff33147bc2fc910 P dd942bee40394ad1a2ca2848b9c8fb1d: Deleting consensus metadata
I20260927 18:46:48.533012 31434 catalog_manager.cc:4841] TS dd942bee40394ad1a2ca2848b9c8fb1d (127.29.159.66:38275): tablet 718e598388784bde9ff33147bc2fc910 (table test-table [id=ac3fb7acf56c4634a394281a9109d042]) successfully deleted
I20260927 18:46:48.533632 31433 catalog_manager.cc:4841] TS 20b47dada65049329e089ff691aff47d (127.29.159.65:40365): tablet 718e598388784bde9ff33147bc2fc910 (table test-table [id=ac3fb7acf56c4634a394281a9109d042]) successfully deleted
Sep 27 18:46:48 dist-test-slave-8tg2 krb5kdc[31403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:46:48.545843 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.533841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.29.159.65:40365)
0927 18:46:48.533952 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.533957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.534257 (+ 300us) server_negotiation.cc:407] Connection header received
0927 18:46:48.534383 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.534386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.534436 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.534522 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:48.535387 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.536213 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.537368 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.537709 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.537812 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.541634 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:48.541645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:48.541670 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:48.541698 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:48.543799 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:48.544373 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:48.544381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:48.544383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:48.544484 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:48.544763 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:48.544766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:48.544768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:48.545071 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:48.545296 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.545492 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.545613 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 18:46:48.546250 31645 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58290
I20260927 18:46:48.557459 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:48.548219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.29.159.126:45821)
0927 18:46:48.548343 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:48.548348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:48.548757 (+ 409us) server_negotiation.cc:407] Connection header received
0927 18:46:48.548858 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:48.548861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:48.548912 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:48.548994 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:48.549754 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.550571 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.551720 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:48.552076 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:48.552189 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:48.553812 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:48.553816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:48.553819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:48.553848 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:48.555684 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:48.556154 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:48.556157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:48.556160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:48.556248 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:48.556534 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:48.556538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:48.556540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:48.556758 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:48.556882 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:48.557167 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 18:46:48.557291 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 18:46:48.558179 31448 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46294
I20260927 18:46:48.559679 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 31520
I20260927 18:46:48.566869 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 31660
I20260927 18:46:48.575537 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 31800
I20260927 18:46:48.582716 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 31416
2026-09-27T18:46:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2853 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:48 dist-test-slave-8tg2 krb5kdc[32022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:48 dist-test-slave-8tg2 krb5kdc[32022](info): set up 2 sockets
Sep 27 18:46:48 dist-test-slave-8tg2 krb5kdc[32022](info): commencing operation
krb5kdc: starting...
W20260927 18:46:49.627274 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:46:49 dist-test-slave-8tg2 krb5kdc[32022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:46:49.750561 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:46685
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:46685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:49.861054 32035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:49.861361 32035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:49.861404 32035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:49.865355 32035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:49.865410 32035 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:46:49.865427 32035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:49.865445 32035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:49.865461 32035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:49.865509 32035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:49.870849 32035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:46685
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:46685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:49.871981 32035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:49.873049 32035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:49.881518 32040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:49.881608 32043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:49.881696 32035 server_base.cc:1034] running on GCE node
W20260927 18:46:49.881628 32041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:49.882434 32035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:49.883680 32035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:49.884845 32035 hybrid_clock.cc:648] HybridClock initialized: now 1790534809884824 us; error 31 us; skew 500 ppm
Sep 27 18:46:49 dist-test-slave-8tg2 krb5kdc[32022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534809, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:49.892709 32035 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:46:49.893725 32035 webserver.cc:460] Webserver started at http://127.29.159.126:43761/ using document root <none> and password file <none>
I20260927 18:46:49.894385 32035 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:49.894451 32035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:49.894697 32035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:49.896942 32035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "b934b2797c6841bc984b6115acfd8ec9"
format_stamp: "Formatted at 2026-09-27 18:46:49 on dist-test-slave-8tg2"
server_key: "af56de1323a61a277e7179aa023c75d0"
server_key_iv: "f182fcd370bd035a3e48d5a816eed286"
server_key_version: "encryptionkey@0"
I20260927 18:46:49.897545 32035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "b934b2797c6841bc984b6115acfd8ec9"
format_stamp: "Formatted at 2026-09-27 18:46:49 on dist-test-slave-8tg2"
server_key: "af56de1323a61a277e7179aa023c75d0"
server_key_iv: "f182fcd370bd035a3e48d5a816eed286"
server_key_version: "encryptionkey@0"
I20260927 18:46:49.901726 32035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:46:49.904567 32050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:49.905917 32035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:49.906044 32035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "b934b2797c6841bc984b6115acfd8ec9"
format_stamp: "Formatted at 2026-09-27 18:46:49 on dist-test-slave-8tg2"
server_key: "af56de1323a61a277e7179aa023c75d0"
server_key_iv: "f182fcd370bd035a3e48d5a816eed286"
server_key_version: "encryptionkey@0"
I20260927 18:46:49.906159 32035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:49.917339 32035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:49.920306 32035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:49.920511 32035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:49.928464 32035 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:46685
I20260927 18:46:49.928493 32112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:46685 every 8 connection(s)
I20260927 18:46:49.929410 32035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:49.932205 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:49.936674 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 32035
I20260927 18:46:49.936802 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:46:49.937062 30333 external_mini_cluster.cc:1428] Setting key 857cf439098c300d545b538028165ffa
I20260927 18:46:49.938100 32113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9: Bootstrap starting.
I20260927 18:46:49.940492 32113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:49.941232 32113 log.cc:824] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:49.943151 32113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9: No bootstrap required, opened a new log
Sep 27 18:46:49 dist-test-slave-8tg2 krb5kdc[32022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:49.947332 32113 raft_consensus.cc:348] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b934b2797c6841bc984b6115acfd8ec9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46685 } }
I20260927 18:46:49.947665 32113 raft_consensus.cc:374] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:49.947731 32113 raft_consensus.cc:729] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b934b2797c6841bc984b6115acfd8ec9, State: Initialized, Role: FOLLOWER
I20260927 18:46:49.948326 32113 consensus_queue.cc:260] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b934b2797c6841bc984b6115acfd8ec9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46685 } }
I20260927 18:46:49.948522 32113 raft_consensus.cc:388] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:49.948601 32113 raft_consensus.cc:482] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:49.948701 32113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:49.949636 32113 raft_consensus.cc:504] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b934b2797c6841bc984b6115acfd8ec9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46685 } }
I20260927 18:46:49.950006 32113 leader_election.cc:302] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b934b2797c6841bc984b6115acfd8ec9; no voters:
I20260927 18:46:49.950397 32113 leader_election.cc:288] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:49.950502 32118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:49.950732 32118 raft_consensus.cc:686] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [term 1 LEADER]: Becoming Leader. State: Replica: b934b2797c6841bc984b6115acfd8ec9, State: Running, Role: LEADER
I20260927 18:46:49.951043 32118 consensus_queue.cc:237] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b934b2797c6841bc984b6115acfd8ec9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46685 } }
I20260927 18:46:49.951813 32113 sys_catalog.cc:564] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:49.952878 32120 sys_catalog.cc:455] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b934b2797c6841bc984b6115acfd8ec9. Latest consensus state: current_term: 1 leader_uuid: "b934b2797c6841bc984b6115acfd8ec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b934b2797c6841bc984b6115acfd8ec9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46685 } } }
I20260927 18:46:49.953011 32120 sys_catalog.cc:458] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:49.953145 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:49.938543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.29.159.126:46685)
0927 18:46:49.939000 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:49.939015 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:49.939051 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:46:49.939919 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:49.939940 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:49.940285 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:49.940618 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:49.941405 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:49.942557 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:49.943548 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:49.944020 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:49.944193 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:49.947264 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:49.947271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:49.947287 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:49.947317 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:49.949296 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:49.949770 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:49.949777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:49.949784 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:49.949905 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:49.950188 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:49.950192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:49.950194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:49.950761 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:49.951585 (+ 824us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:49.951741 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 18:46:49.952059 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":149,"threads_started":1}
I20260927 18:46:49.953328 32119 sys_catalog.cc:455] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b934b2797c6841bc984b6115acfd8ec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b934b2797c6841bc984b6115acfd8ec9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46685 } } }
I20260927 18:46:49.953406 32119 sys_catalog.cc:458] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:49.953661 32127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:49.956630 32127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:49.961726 32127 catalog_manager.cc:1347] Generated new cluster ID: 193bd20a38f7413385ca66f12fcb78bc
I20260927 18:46:49.961805 32127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:49.988263 32127 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:49.989270 32127 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:49.999982 32127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9: Generated new TSK 0
I20260927 18:46:50.000761 32127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:46:50.032828 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:46685
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:50.145248 32139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:50.145556 32139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:50.145635 32139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:50.149753 32139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:50.149843 32139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:50.149919 32139 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:46:50.149959 32139 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:46:50.150025 32139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:46:50.155769 32139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.159.126:46685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:50.156901 32139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:50.158387 32139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:50.169965 32144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:50.170073 32139 server_base.cc:1034] running on GCE node
W20260927 18:46:50.169997 32147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:50.170013 32145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:50.170610 32139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:50.171319 32139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:50.172504 32139 hybrid_clock.cc:648] HybridClock initialized: now 1790534810172483 us; error 33 us; skew 500 ppm
Sep 27 18:46:50 dist-test-slave-8tg2 krb5kdc[32022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534810, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:50.175362 32139 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:46:50.176249 32139 webserver.cc:460] Webserver started at http://127.29.159.65:33341/ using document root <none> and password file <none>
I20260927 18:46:50.176820 32139 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:50.176882 32139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:50.177093 32139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:50.179260 32139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "78b172c2925c4e99a334ff57de37e502"
format_stamp: "Formatted at 2026-09-27 18:46:50 on dist-test-slave-8tg2"
server_key: "64cb07d479ab01f0d280e6b2e54d8e67"
server_key_iv: "65504025f72e4c6429d7f907545976cc"
server_key_version: "encryptionkey@0"
I20260927 18:46:50.179890 32139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "78b172c2925c4e99a334ff57de37e502"
format_stamp: "Formatted at 2026-09-27 18:46:50 on dist-test-slave-8tg2"
server_key: "64cb07d479ab01f0d280e6b2e54d8e67"
server_key_iv: "65504025f72e4c6429d7f907545976cc"
server_key_version: "encryptionkey@0"
I20260927 18:46:50.183930 32139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:46:50.186468 32154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:50.187803 32139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:50.187928 32139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "78b172c2925c4e99a334ff57de37e502"
format_stamp: "Formatted at 2026-09-27 18:46:50 on dist-test-slave-8tg2"
server_key: "64cb07d479ab01f0d280e6b2e54d8e67"
server_key_iv: "65504025f72e4c6429d7f907545976cc"
server_key_version: "encryptionkey@0"
I20260927 18:46:50.188040 32139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:50.208623 32139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:50.212052 32139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:50.212248 32139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:50.213689 32139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:50.213760 32139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:50.213828 32139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:50.213867 32139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:46:50 dist-test-slave-8tg2 krb5kdc[32022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534810, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:50.224805 32139 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:38795
I20260927 18:46:50.224879 32269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:38795 every 8 connection(s)
I20260927 18:46:50.225713 32139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:50.228291 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.214636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:41445 (local address 127.29.159.126:46685)
0927 18:46:50.214786 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.214789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.215941 (+ 1152us) server_negotiation.cc:407] Connection header received
0927 18:46:50.217145 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.217149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.217224 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.217320 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:50.219117 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.219957 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.221219 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.221470 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.221583 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.224330 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:50.224334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:50.224337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:50.224374 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:50.226079 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.226674 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.226681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.226683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.226775 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.227100 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.227104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.227106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.227308 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:50.227414 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.227940 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.228086 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:46:50.229061 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.215241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46685 (local address 127.29.159.65:41445)
0927 18:46:50.215791 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.215835 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.216901 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.217543 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.217556 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.218107 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:50.218933 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.218958 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.220092 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.220096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.221082 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.221092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.221623 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.221627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.221842 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.222492 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:50.222508 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:50.224132 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:50.226234 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.226239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.226254 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.226567 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.226867 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.226872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.226874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.227006 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.227439 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:50.227449 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:50.227710 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.227898 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.228256 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":168,"threads_started":1}
I20260927 18:46:50.229867 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 32139
I20260927 18:46:50.230005 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:50.230295 30333 external_mini_cluster.cc:1428] Setting key 4ee12dfe53812bdaf8aacc98cf67a44d
I20260927 18:46:50.230474 32270 heartbeater.cc:347] Connected to a master server at 127.29.159.126:46685
I20260927 18:46:50.230614 32270 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:50.231177 32270 heartbeater.cc:511] Master 127.29.159.126:46685 requested a full tablet report, sending...
I20260927 18:46:50.233989 32066 ts_manager.cc:194] Registered new tserver with Master: 78b172c2925c4e99a334ff57de37e502 (127.29.159.65:38795)
I20260927 18:46:50.235107 32066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:41445
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:46:50.243137 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.232692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46685 (local address 127.29.159.65:40975)
0927 18:46:50.236089 (+ 3397us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.236100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.236235 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.236470 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.236473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.236678 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:50.236782 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.236787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.237507 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.237511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.238073 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.238080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.238803 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.238806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.238880 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.239401 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:50.239413 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:50.239785 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:50.241697 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.241699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.241701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.241895 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.242170 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.242172 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.242173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.242248 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.242689 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:50.242691 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:50.242770 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.242863 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.242979 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3317,"thread_start_us":89,"threads_started":1}
I20260927 18:46:50.243208 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.232751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:40975 (local address 127.29.159.126:46685)
0927 18:46:50.232857 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.232865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.236175 (+ 3310us) server_negotiation.cc:407] Connection header received
0927 18:46:50.236326 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.236330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.236376 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.236454 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:50.236892 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.237390 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.238272 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.238683 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.238811 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.239923 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:50.239929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:50.239935 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:50.239968 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:50.241583 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.241994 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.241998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.242000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.242082 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.242337 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.242342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.242345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.242591 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:50.242700 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.242985 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.243090 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:46:50.262596 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:46685
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:50.378136 32279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:50.378417 32279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:50.378499 32279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:50.382781 32279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:50.382848 32279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:50.382913 32279 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:46:50.382952 32279 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:46:50.383013 32279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:46:50.388036 32279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.159.126:46685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:50.389166 32279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:50.390148 32279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:50.399487 32287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:50.399574 32284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:50.399518 32285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:50.399695 32279 server_base.cc:1034] running on GCE node
I20260927 18:46:50.400131 32279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:50.400759 32279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:50.401947 32279 hybrid_clock.cc:648] HybridClock initialized: now 1790534810401934 us; error 25 us; skew 500 ppm
Sep 27 18:46:50 dist-test-slave-8tg2 krb5kdc[32022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534810, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:50.404758 32279 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:46:50.405568 32279 webserver.cc:460] Webserver started at http://127.29.159.66:44427/ using document root <none> and password file <none>
I20260927 18:46:50.406080 32279 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:50.406149 32279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:50.406313 32279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:50.408372 32279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "3ccf86ab155a4e1c8f929bdd995431d6"
format_stamp: "Formatted at 2026-09-27 18:46:50 on dist-test-slave-8tg2"
server_key: "8ff661ba675724a73c6f4e2a3040a29f"
server_key_iv: "01837095cb63ae889fa0b02a4b29ea5b"
server_key_version: "encryptionkey@0"
I20260927 18:46:50.408891 32279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "3ccf86ab155a4e1c8f929bdd995431d6"
format_stamp: "Formatted at 2026-09-27 18:46:50 on dist-test-slave-8tg2"
server_key: "8ff661ba675724a73c6f4e2a3040a29f"
server_key_iv: "01837095cb63ae889fa0b02a4b29ea5b"
server_key_version: "encryptionkey@0"
I20260927 18:46:50.412554 32279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:46:50.414848 32294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:50.416121 32279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:50.416244 32279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "3ccf86ab155a4e1c8f929bdd995431d6"
format_stamp: "Formatted at 2026-09-27 18:46:50 on dist-test-slave-8tg2"
server_key: "8ff661ba675724a73c6f4e2a3040a29f"
server_key_iv: "01837095cb63ae889fa0b02a4b29ea5b"
server_key_version: "encryptionkey@0"
I20260927 18:46:50.416347 32279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:50.446959 32279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:50.450363 32279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:50.450579 32279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:50.452412 32279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:50.452482 32279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:50.452548 32279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:50.452584 32279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:46:50 dist-test-slave-8tg2 krb5kdc[32022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534810, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:50.463389 32279 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:41305
I20260927 18:46:50.463500 32409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:41305 every 8 connection(s)
I20260927 18:46:50.464473 32279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:50.466709 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.453079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:44757 (local address 127.29.159.126:46685)
0927 18:46:50.453236 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.453243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.454743 (+ 1500us) server_negotiation.cc:407] Connection header received
0927 18:46:50.456155 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.456158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.456204 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.456279 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:50.458053 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.458766 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.460086 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.460422 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.460553 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.463159 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:50.463164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:50.463167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:50.463201 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:50.464737 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.465361 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.465367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.465372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.465471 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.465813 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.465815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.465816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.465991 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:50.466071 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.466496 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.466591 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:46:50.467584 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.454032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46685 (local address 127.29.159.66:44757)
0927 18:46:50.454542 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.454612 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.455923 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.456492 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.456505 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.457077 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:50.457876 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.457901 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.458932 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.458936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.459954 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.459969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.460584 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.460587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.460727 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.461353 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:50.461389 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:50.462947 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:50.464861 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.464867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.464883 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.465251 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.465565 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.465568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.465569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.465722 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.466076 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:50.466080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:50.466301 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.466453 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.466716 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":104,"threads_started":1}
I20260927 18:46:50.468708 32410 heartbeater.cc:347] Connected to a master server at 127.29.159.126:46685
I20260927 18:46:50.468837 32410 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:50.468977 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 32279
I20260927 18:46:50.469080 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:50.469352 30333 external_mini_cluster.cc:1428] Setting key a5dc4b904d7d0e8d164564001a6a88b5
I20260927 18:46:50.469364 32410 heartbeater.cc:511] Master 127.29.159.126:46685 requested a full tablet report, sending...
I20260927 18:46:50.470461 32066 ts_manager.cc:194] Registered new tserver with Master: 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305)
I20260927 18:46:50.471190 32066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:44757
I20260927 18:46:50.478984 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.471324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:59729 (local address 127.29.159.126:46685)
0927 18:46:50.471444 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.471450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.471465 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:46:50.471503 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.471506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.471543 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.471644 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:50.472172 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.472863 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.474097 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.474389 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.474471 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.475766 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:50.475769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:50.475771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:50.475791 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:50.477242 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.477724 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.477726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.477728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.477798 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.478094 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.478095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.478097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.478376 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:50.478477 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.478783 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.478878 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:46:50.479071 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.471046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46685 (local address 127.29.159.66:59729)
0927 18:46:50.471283 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.471297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.471401 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.471660 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.471666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.471896 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:50.472043 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.472052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.473018 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.473024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.473963 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.473976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.474505 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.474511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.474641 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.475148 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:50.475167 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:50.475642 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:50.477354 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.477359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.477364 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.477630 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.477908 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.477913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.477918 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.478008 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.478484 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:50.478487 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:50.478598 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.478738 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.478892 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:46:50.496176 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:46685
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:50.629196 32419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:50.629446 32419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:50.629491 32419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:50.633482 32419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:50.633538 32419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:50.633576 32419 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:46:50.633590 32419 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:46:50.633657 32419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:46:50.639319 32419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.159.126:46685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:50.640362 32419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:50.641383 32419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:50.650892 32425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:50.650892 32427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:50.650946 32424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:50.650998 32419 server_base.cc:1034] running on GCE node
I20260927 18:46:50.651667 32419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:50.652305 32419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:50.653455 32419 hybrid_clock.cc:648] HybridClock initialized: now 1790534810653392 us; error 79 us; skew 500 ppm
Sep 27 18:46:50 dist-test-slave-8tg2 krb5kdc[32022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534810, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:50.656199 32419 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:46:50.657069 32419 webserver.cc:460] Webserver started at http://127.29.159.67:39379/ using document root <none> and password file <none>
I20260927 18:46:50.657603 32419 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:50.657713 32419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:50.657949 32419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:50.659976 32419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e"
format_stamp: "Formatted at 2026-09-27 18:46:50 on dist-test-slave-8tg2"
server_key: "b0007cb34b11bf2b4ad15d44f30b2c7f"
server_key_iv: "39c8395c1254636fae734a3b246a33ba"
server_key_version: "encryptionkey@0"
I20260927 18:46:50.660519 32419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e"
format_stamp: "Formatted at 2026-09-27 18:46:50 on dist-test-slave-8tg2"
server_key: "b0007cb34b11bf2b4ad15d44f30b2c7f"
server_key_iv: "39c8395c1254636fae734a3b246a33ba"
server_key_version: "encryptionkey@0"
I20260927 18:46:50.664294 32419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:46:50.666632 32434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:50.668040 32419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:50.668184 32419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e"
format_stamp: "Formatted at 2026-09-27 18:46:50 on dist-test-slave-8tg2"
server_key: "b0007cb34b11bf2b4ad15d44f30b2c7f"
server_key_iv: "39c8395c1254636fae734a3b246a33ba"
server_key_version: "encryptionkey@0"
I20260927 18:46:50.668291 32419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:50.687398 32419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:50.690701 32419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:50.690934 32419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:50.692627 32419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:50.692694 32419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:50.692762 32419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:50.692814 32419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:46:50 dist-test-slave-8tg2 krb5kdc[32022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534810, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:50.704154 32419 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:34547
I20260927 18:46:50.704350 32549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:34547 every 8 connection(s)
I20260927 18:46:50.705202 32419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:50.707568 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.693684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:51007 (local address 127.29.159.126:46685)
0927 18:46:50.693820 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.693824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.694962 (+ 1138us) server_negotiation.cc:407] Connection header received
0927 18:46:50.696325 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.696330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.696398 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.696514 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:50.697927 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.698634 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.700219 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.700467 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.700606 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.703594 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:50.703598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:50.703600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:50.703628 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:50.705307 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.705971 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.705975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.705977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.706089 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.706506 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.706509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.706510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.706706 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:50.706790 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.707269 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.707371 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:46:50.708370 32443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.694206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46685 (local address 127.29.159.67:51007)
0927 18:46:50.694805 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.694854 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.696054 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.696714 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.696722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.697099 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:50.697773 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.697788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.698770 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.698773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.700087 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.700098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.700675 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.700680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.700869 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.701551 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:50.701569 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:50.703368 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:50.705453 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.705460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.705476 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.705827 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.706206 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.706209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.706211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.706377 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.706794 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:50.706799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:50.707021 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.707170 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.707480 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":140,"threads_started":1}
I20260927 18:46:50.709561 32550 heartbeater.cc:347] Connected to a master server at 127.29.159.126:46685
I20260927 18:46:50.709717 32550 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:50.710204 32550 heartbeater.cc:511] Master 127.29.159.126:46685 requested a full tablet report, sending...
I20260927 18:46:50.711225 32067 ts_manager.cc:194] Registered new tserver with Master: ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67:34547)
I20260927 18:46:50.712024 32067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:51007
I20260927 18:46:50.712396 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 32419
I20260927 18:46:50.712514 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:50.712888 30333 external_mini_cluster.cc:1428] Setting key 9a2a5699613b950160fb776ed9210655
I20260927 18:46:50.715637 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:46:50.722339 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.711888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46685 (local address 127.29.159.67:58987)
0927 18:46:50.712173 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.712188 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.712276 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.712607 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.712612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.712884 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:50.713043 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.713066 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.714074 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.714077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.714792 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.714800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.715331 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.715335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.715460 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.715990 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:50.716007 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:50.716472 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:50.719009 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.719012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.719015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.719327 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.719770 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.719774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.719775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.719869 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.721743 (+ 1874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:50.721746 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:50.721876 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.721994 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.722132 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":83,"threads_started":1}
I20260927 18:46:50.722869 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.711993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:58987 (local address 127.29.159.126:46685)
0927 18:46:50.712388 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.712392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.712403 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:46:50.712434 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.712437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.712475 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.712555 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:50.713216 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.713949 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.714894 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.715198 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.715308 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.716632 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:50.716639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:50.716642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:50.716670 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:50.718894 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.719484 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.719487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.719489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.719661 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.719983 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.719989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.719994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.720239 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:50.720353 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.722116 (+ 1763us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.722332 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
Sep 27 18:46:50 dist-test-slave-8tg2 krb5kdc[32022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:46:50 dist-test-slave-8tg2 krb5kdc[32022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:50.742425 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.734606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.29.159.126:46685)
0927 18:46:50.734709 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.734712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.734722 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:46:50.734754 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.734756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.734791 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.734866 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:50.735734 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.736217 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.736989 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.737177 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.737248 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.739733 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:50.739736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:50.739738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:50.739760 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:50.740983 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.741441 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.741444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.741445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.741527 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.741763 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.741765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.741766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.741929 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:50.742029 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.742212 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.742311 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:46:50.745211 32067 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:50.748420 32067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:50.763002 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.758056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.29.159.66:41305)
0927 18:46:50.758483 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.758493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.758512 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:46:50.758659 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.758665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.758830 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.758958 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:50.759338 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.760318 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.761723 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.762255 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.762390 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.762571 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.762729 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.762838 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":115,"threads_started":1}
I20260927 18:46:50.763022 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.757543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:41305 (local address 127.0.0.1:48072)
0927 18:46:50.758419 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.758433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.758532 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.759092 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.759096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.759116 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:50.759215 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.759221 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.760440 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.760443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.761588 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.761598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.762388 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.762392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.762505 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.762540 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.762691 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.762903 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":62,"thread_start_us":88,"threads_started":1}
I20260927 18:46:50.763346 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.757245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.29.159.65:38795)
0927 18:46:50.757546 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.757553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.757580 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:46:50.757667 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.757673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.757908 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.758055 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:50.758597 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.759654 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.761456 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.762258 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.762376 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.762548 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.763081 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.763182 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
I20260927 18:46:50.763025 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.757009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:34547 (local address 127.0.0.1:39398)
0927 18:46:50.757737 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.757751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.757834 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.758826 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.758829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.758849 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:50.758954 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.758960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.760232 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.760236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.761552 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.761563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.762545 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.762549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.762653 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.762680 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.762733 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.762853 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":93,"thread_start_us":94,"threads_started":1}
I20260927 18:46:50.763551 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.758027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.29.159.67:34547)
0927 18:46:50.758349 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.758361 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.758386 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:46:50.758462 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.758467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.758682 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.758844 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:50.759053 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.760076 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.761690 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.762411 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.762832 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.762984 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.763036 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.763349 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":124,"threads_started":1}
I20260927 18:46:50.763741 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.756825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:38795 (local address 127.0.0.1:44424)
0927 18:46:50.757260 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.757312 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.757439 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.758282 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.758287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.758315 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:50.758459 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.758469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.759765 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.759768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.761328 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.761337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.762596 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.762600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.762680 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.762697 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.762760 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.762846 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":115,"thread_start_us":84,"threads_started":1}
I20260927 18:46:50.766317 32344 tablet_service.cc:1467] Processing CreateTablet for tablet 0618b41f13e54cb795e70187e125cb5b (DEFAULT_TABLE table=test-table [id=3f7d7e2d6a7b414e8727818182dbf55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:50.766633 32204 tablet_service.cc:1467] Processing CreateTablet for tablet 0618b41f13e54cb795e70187e125cb5b (DEFAULT_TABLE table=test-table [id=3f7d7e2d6a7b414e8727818182dbf55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:50.767557 32344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0618b41f13e54cb795e70187e125cb5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:50.767264 32484 tablet_service.cc:1467] Processing CreateTablet for tablet 0618b41f13e54cb795e70187e125cb5b (DEFAULT_TABLE table=test-table [id=3f7d7e2d6a7b414e8727818182dbf55b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:50.767732 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0618b41f13e54cb795e70187e125cb5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:50.768232 32484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0618b41f13e54cb795e70187e125cb5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:50.774267 32573 tablet_bootstrap.cc:492] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: Bootstrap starting.
I20260927 18:46:50.775718 32574 tablet_bootstrap.cc:492] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: Bootstrap starting.
I20260927 18:46:50.776957 32573 tablet_bootstrap.cc:654] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:50.777802 32574 tablet_bootstrap.cc:654] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:50.777875 32573 log.cc:824] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:50.778684 32574 log.cc:824] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:50.779399 32575 tablet_bootstrap.cc:492] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: Bootstrap starting.
I20260927 18:46:50.780107 32573 tablet_bootstrap.cc:492] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: No bootstrap required, opened a new log
I20260927 18:46:50.780323 32573 ts_tablet_manager.cc:1397] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:46:50.781100 32574 tablet_bootstrap.cc:492] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: No bootstrap required, opened a new log
I20260927 18:46:50.781307 32574 ts_tablet_manager.cc:1397] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:46:50.781634 32575 tablet_bootstrap.cc:654] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:50.782541 32575 log.cc:824] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:50.784696 32575 tablet_bootstrap.cc:492] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: No bootstrap required, opened a new log
I20260927 18:46:50.784896 32575 ts_tablet_manager.cc:1397] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 18:46:50.784790 32574 raft_consensus.cc:348] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } }
I20260927 18:46:50.785113 32574 raft_consensus.cc:374] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:50.784976 32573 raft_consensus.cc:348] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } }
I20260927 18:46:50.785176 32574 raft_consensus.cc:729] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed94647a16dd4fe8a74bce6c4fa0b58e, State: Initialized, Role: FOLLOWER
I20260927 18:46:50.785240 32573 raft_consensus.cc:374] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:50.785308 32573 raft_consensus.cc:729] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b172c2925c4e99a334ff57de37e502, State: Initialized, Role: FOLLOWER
I20260927 18:46:50.785703 32574 consensus_queue.cc:260] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } }
I20260927 18:46:50.785827 32573 consensus_queue.cc:260] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } }
I20260927 18:46:50.786734 32574 ts_tablet_manager.cc:1428] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:46:50.786762 32573 ts_tablet_manager.cc:1428] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 18:46:50.787109 32550 heartbeater.cc:503] Master 127.29.159.126:46685 was elected leader, sending a full tablet report...
I20260927 18:46:50.787106 32270 heartbeater.cc:503] Master 127.29.159.126:46685 was elected leader, sending a full tablet report...
I20260927 18:46:50.788867 32575 raft_consensus.cc:348] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } }
I20260927 18:46:50.789149 32575 raft_consensus.cc:374] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:50.789238 32575 raft_consensus.cc:729] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf86ab155a4e1c8f929bdd995431d6, State: Initialized, Role: FOLLOWER
I20260927 18:46:50.789719 32575 consensus_queue.cc:260] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } }
I20260927 18:46:50.790450 32410 heartbeater.cc:503] Master 127.29.159.126:46685 was elected leader, sending a full tablet report...
I20260927 18:46:50.790943 32575 ts_tablet_manager.cc:1428] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260927 18:46:50.842803 32579 raft_consensus.cc:482] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:50.843037 32579 raft_consensus.cc:504] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } }
I20260927 18:46:50.844307 32579 leader_election.cc:288] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b172c2925c4e99a334ff57de37e502 (127.29.159.65:38795), 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305)
I20260927 18:46:50.848272 32443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.844565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:41305 (local address 127.29.159.67:46213)
0927 18:46:50.844752 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.844768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.844864 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.845172 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.845175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.845193 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:50.845355 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.845364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.846148 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.846150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.847141 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.847152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.847951 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.847953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.848032 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.848052 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.848113 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.848170 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 18:46:50.848345 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.844733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:46213 (local address 127.29.159.66:41305)
0927 18:46:50.844922 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.844926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.844940 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:46:50.844988 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.844990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.845028 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.845145 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:50.845480 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.846042 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.847281 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.847839 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.847960 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.848013 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.848142 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.848216 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:46:50.848942 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.844567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:38795 (local address 127.29.159.67:40995)
0927 18:46:50.844990 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.845000 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.845086 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.845387 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.845393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.845416 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:50.845867 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.845873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.846660 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.846664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.847631 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.847638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.848604 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.848608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.848707 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.848724 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.848785 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.848834 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":30,"thread_start_us":151,"threads_started":1}
I20260927 18:46:50.849013 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.844615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:40995 (local address 127.29.159.65:38795)
0927 18:46:50.844741 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.844745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.845087 (+ 342us) server_negotiation.cc:407] Connection header received
0927 18:46:50.845209 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.845211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.845291 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.845355 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:50.845988 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.846553 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.847745 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.848372 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.848495 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.848547 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.848820 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.848897 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:46:50.848976 32364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0618b41f13e54cb795e70187e125cb5b" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" is_pre_election: true
I20260927 18:46:50.849323 32364 raft_consensus.cc:2454] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 0.
I20260927 18:46:50.849573 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0618b41f13e54cb795e70187e125cb5b" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b172c2925c4e99a334ff57de37e502" is_pre_election: true
I20260927 18:46:50.849866 32224 raft_consensus.cc:2454] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 0.
I20260927 18:46:50.849862 32437 leader_election.cc:302] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccf86ab155a4e1c8f929bdd995431d6, ed94647a16dd4fe8a74bce6c4fa0b58e; no voters:
I20260927 18:46:50.850157 32579 raft_consensus.cc:2781] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:50.850270 32579 raft_consensus.cc:482] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:50.850332 32579 raft_consensus.cc:3037] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:50.851405 32579 raft_consensus.cc:504] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } }
I20260927 18:46:50.851845 32579 leader_election.cc:288] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 election: Requested vote from peers 78b172c2925c4e99a334ff57de37e502 (127.29.159.65:38795), 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305)
I20260927 18:46:50.852195 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0618b41f13e54cb795e70187e125cb5b" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b172c2925c4e99a334ff57de37e502"
I20260927 18:46:50.852200 32364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0618b41f13e54cb795e70187e125cb5b" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf86ab155a4e1c8f929bdd995431d6"
I20260927 18:46:50.852320 32224 raft_consensus.cc:3037] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:50.852329 32364 raft_consensus.cc:3037] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:50.853683 32364 raft_consensus.cc:2454] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 1.
I20260927 18:46:50.853711 32224 raft_consensus.cc:2454] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 1.
I20260927 18:46:50.854054 32435 leader_election.cc:302] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ccf86ab155a4e1c8f929bdd995431d6, 78b172c2925c4e99a334ff57de37e502, ed94647a16dd4fe8a74bce6c4fa0b58e; no voters:
I20260927 18:46:50.854260 32579 raft_consensus.cc:2781] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:50.854503 32579 raft_consensus.cc:686] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 LEADER]: Becoming Leader. State: Replica: ed94647a16dd4fe8a74bce6c4fa0b58e, State: Running, Role: LEADER
I20260927 18:46:50.854883 32579 consensus_queue.cc:237] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } }
I20260927 18:46:50.858067 32065 catalog_manager.cc:5495] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e reported cstate change: term changed from 0 to 1, leader changed from <none> to ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67). New cstate: current_term: 1 leader_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:46:50 dist-test-slave-8tg2 krb5kdc[32022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534809, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:50.909693 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.901321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.29.159.126:46685)
0927 18:46:50.901484 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:50.901489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:50.901665 (+ 176us) server_negotiation.cc:407] Connection header received
0927 18:46:50.901775 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:50.901777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:50.901830 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:50.901907 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:50.902611 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.903116 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.904021 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.904221 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.904314 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.906529 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:50.906533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:50.906535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:50.906556 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:50.907874 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.908425 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.908427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.908429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.908498 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:50.908724 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:50.908726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:50.908727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:50.908902 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:50.908977 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:50.909371 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 18:46:50.909542 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":77}
I20260927 18:46:50.909718 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:50.901241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46685 (local address 127.0.0.1:42068)
0927 18:46:50.901568 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:50.901581 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:50.901695 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:50.901942 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:50.901945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:50.902413 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:50.902510 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.902516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.903210 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.903212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.903923 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:50.903933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:50.904338 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:50.904341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:50.904406 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:50.904944 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:50.904955 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:50.906404 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:50.907983 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.907991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.907997 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.908340 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.908571 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:50.908573 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:50.908574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:50.908650 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:50.909018 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:50.909024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:50.909183 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 18:46:50.909308 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 18:46:50.909469 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":160,"threads_started":1}
I20260927 18:46:50.913796 32065 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:42068:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:46:50.914491 32065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:50.919250 32204 tablet_service.cc:1467] Processing CreateTablet for tablet 163c1980596343a887f076f05f1c00eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b6aa9075508b48b9bbf4af0f7b85b06e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:46:50.919256 32484 tablet_service.cc:1467] Processing CreateTablet for tablet 163c1980596343a887f076f05f1c00eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b6aa9075508b48b9bbf4af0f7b85b06e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:46:50.919250 32344 tablet_service.cc:1467] Processing CreateTablet for tablet 163c1980596343a887f076f05f1c00eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b6aa9075508b48b9bbf4af0f7b85b06e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:46:50.920003 32484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 163c1980596343a887f076f05f1c00eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:50.920003 32344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 163c1980596343a887f076f05f1c00eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:50.920009 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 163c1980596343a887f076f05f1c00eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:50.922616 32573 tablet_bootstrap.cc:492] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502: Bootstrap starting.
I20260927 18:46:50.922869 32574 tablet_bootstrap.cc:492] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: Bootstrap starting.
I20260927 18:46:50.922986 32575 tablet_bootstrap.cc:492] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6: Bootstrap starting.
I20260927 18:46:50.923748 32574 tablet_bootstrap.cc:654] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:50.923761 32573 tablet_bootstrap.cc:654] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:50.924423 32575 tablet_bootstrap.cc:654] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:50.924963 32574 tablet_bootstrap.cc:492] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: No bootstrap required, opened a new log
I20260927 18:46:50.925060 32574 ts_tablet_manager.cc:1397] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:46:50.925621 32575 tablet_bootstrap.cc:492] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6: No bootstrap required, opened a new log
I20260927 18:46:50.925592 32574 raft_consensus.cc:348] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:50.925701 32574 raft_consensus.cc:374] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:50.925731 32575 ts_tablet_manager.cc:1397] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:46:50.925741 32574 raft_consensus.cc:729] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed94647a16dd4fe8a74bce6c4fa0b58e, State: Initialized, Role: FOLLOWER
I20260927 18:46:50.925796 32573 tablet_bootstrap.cc:492] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502: No bootstrap required, opened a new log
I20260927 18:46:50.925882 32573 ts_tablet_manager.cc:1397] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:46:50.925928 32574 consensus_queue.cc:260] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:50.926244 32574 ts_tablet_manager.cc:1428] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 18:46:50.926407 32573 raft_consensus.cc:348] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:50.926540 32573 raft_consensus.cc:374] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:50.926512 32579 tablet_replica.cc:439] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } }
I20260927 18:46:50.926580 32573 raft_consensus.cc:729] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b172c2925c4e99a334ff57de37e502, State: Initialized, Role: FOLLOWER
I20260927 18:46:50.926646 32579 tablet_replica.cc:442] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:50.926702 32573 consensus_queue.cc:260] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:50.926254 32575 raft_consensus.cc:348] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:50.926855 32575 raft_consensus.cc:374] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:50.926910 32575 raft_consensus.cc:729] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf86ab155a4e1c8f929bdd995431d6, State: Initialized, Role: FOLLOWER
I20260927 18:46:50.926976 32573 ts_tablet_manager.cc:1428] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:46:50.927090 32580 tablet_replica.cc:439] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } }
I20260927 18:46:50.927203 32580 tablet_replica.cc:442] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:50.927546 32575 consensus_queue.cc:260] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:50.927846 32581 tablet_replica.cc:439] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } }
I20260927 18:46:50.927982 32581 tablet_replica.cc:442] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:50.927834 32575 ts_tablet_manager.cc:1428] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260927 18:46:50.956491 32551 tablet.cc:2369] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:46:50.969561 32411 tablet.cc:2369] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:46:50.977088 32271 tablet.cc:2369] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:46:51.001832 32579 raft_consensus.cc:482] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:51.001966 32579 raft_consensus.cc:504] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.002382 32579 leader_election.cc:288] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b172c2925c4e99a334ff57de37e502 (127.29.159.65:38795), 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305)
I20260927 18:46:51.002890 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "163c1980596343a887f076f05f1c00eb" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b172c2925c4e99a334ff57de37e502" is_pre_election: true
I20260927 18:46:51.002892 32364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "163c1980596343a887f076f05f1c00eb" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" is_pre_election: true
I20260927 18:46:51.003027 32224 raft_consensus.cc:2454] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 0.
I20260927 18:46:51.003062 32364 raft_consensus.cc:2454] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 0.
I20260927 18:46:51.003341 32435 leader_election.cc:302] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b172c2925c4e99a334ff57de37e502, ed94647a16dd4fe8a74bce6c4fa0b58e; no voters:
I20260927 18:46:51.003556 32579 raft_consensus.cc:2781] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:51.003624 32579 raft_consensus.cc:482] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:51.003654 32579 raft_consensus.cc:3037] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:51.004690 32579 raft_consensus.cc:504] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.005046 32579 leader_election.cc:288] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 election: Requested vote from peers 78b172c2925c4e99a334ff57de37e502 (127.29.159.65:38795), 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305)
I20260927 18:46:51.005354 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "163c1980596343a887f076f05f1c00eb" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b172c2925c4e99a334ff57de37e502"
I20260927 18:46:51.005441 32364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "163c1980596343a887f076f05f1c00eb" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf86ab155a4e1c8f929bdd995431d6"
I20260927 18:46:51.005502 32224 raft_consensus.cc:3037] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:51.005573 32364 raft_consensus.cc:3037] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:51.006539 32224 raft_consensus.cc:2454] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 1.
I20260927 18:46:51.006567 32364 raft_consensus.cc:2454] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 1.
I20260927 18:46:51.006860 32435 leader_election.cc:302] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b172c2925c4e99a334ff57de37e502, ed94647a16dd4fe8a74bce6c4fa0b58e; no voters:
I20260927 18:46:51.007085 32579 raft_consensus.cc:2781] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:51.007157 32579 raft_consensus.cc:686] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 LEADER]: Becoming Leader. State: Replica: ed94647a16dd4fe8a74bce6c4fa0b58e, State: Running, Role: LEADER
I20260927 18:46:51.007288 32579 consensus_queue.cc:237] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.007965 32583 tablet_replica.cc:439] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: TxnStatusTablet state changed. Reason: New leader ed94647a16dd4fe8a74bce6c4fa0b58e. Latest consensus state: current_term: 1 leader_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } }
I20260927 18:46:51.008080 32583 tablet_replica.cc:442] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:46:51.008399 32595 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:46:51.008651 32595 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:46:51.008971 32065 catalog_manager.cc:5495] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e reported cstate change: term changed from 0 to 1, leader changed from <none> to ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67). New cstate: current_term: 1 leader_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } health_report { overall_health: HEALTHY } } }
I20260927 18:46:51.035352 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.031287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.29.159.67:34547)
0927 18:46:51.031417 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:51.031452 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:51.031469 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:46:51.031685 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:51.031687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:51.031741 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:51.031843 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:51.032113 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.032683 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.033887 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.034097 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.034228 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.034544 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:51.034956 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:51.035041 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:51.035192 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 18:46:51.035245 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:46:51.035384 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.031126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:34547 (local address 127.0.0.1:39412)
0927 18:46:51.031360 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:51.031376 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:51.031532 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:51.031868 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:51.031871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:51.031879 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:46:51.031992 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.032001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.032796 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.032799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.033775 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.033785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.034234 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.034238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.034362 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.034415 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:46:51.035049 (+ 634us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:46:51.035063 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 18:46:51.035136 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 18:46:51.035226 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 18:46:51.041591 32364 raft_consensus.cc:1261] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Refusing update from remote peer ed94647a16dd4fe8a74bce6c4fa0b58e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:51.041761 32224 raft_consensus.cc:1261] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Refusing update from remote peer ed94647a16dd4fe8a74bce6c4fa0b58e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:51.042358 32583 consensus_queue.cc:1035] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:51.042569 32579 consensus_queue.cc:1035] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:51.046104 32580 tablet_replica.cc:439] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502: TxnStatusTablet state changed. Reason: New leader ed94647a16dd4fe8a74bce6c4fa0b58e. Latest consensus state: current_term: 1 leader_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } }
I20260927 18:46:51.046248 32580 tablet_replica.cc:442] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:51.047082 32581 tablet_replica.cc:439] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6: TxnStatusTablet state changed. Reason: New leader ed94647a16dd4fe8a74bce6c4fa0b58e. Latest consensus state: current_term: 1 leader_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } }
I20260927 18:46:51.047209 32581 tablet_replica.cc:442] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:51.049850 32579 tablet_replica.cc:439] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } }
I20260927 18:46:51.049973 32579 tablet_replica.cc:442] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:46:51.050166 32580 tablet_replica.cc:439] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } }
I20260927 18:46:51.050276 32580 tablet_replica.cc:442] T 163c1980596343a887f076f05f1c00eb P 78b172c2925c4e99a334ff57de37e502: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:51.052096 32598 mvcc.cc:205] Tried to move back new op lower bound from 7334030586017611776 to 7334030585887055872. Current Snapshot: MvccSnapshot[applied={T|T < 7334030586017611776}]
I20260927 18:46:51.052502 32599 mvcc.cc:205] Tried to move back new op lower bound from 7334030586017611776 to 7334030585887055872. Current Snapshot: MvccSnapshot[applied={T|T < 7334030586017611776}]
I20260927 18:46:51.052639 32583 tablet_replica.cc:439] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } }
I20260927 18:46:51.052821 32583 tablet_replica.cc:442] T 163c1980596343a887f076f05f1c00eb P ed94647a16dd4fe8a74bce6c4fa0b58e: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:46:51.053895 32581 tablet_replica.cc:439] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } } }
I20260927 18:46:51.054133 32581 tablet_replica.cc:442] T 163c1980596343a887f076f05f1c00eb P 3ccf86ab155a4e1c8f929bdd995431d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:46:51.064343 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.060628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.29.159.67:34547)
0927 18:46:51.060768 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:51.060771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:51.060784 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:51.060883 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:51.060885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:51.060924 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:51.060985 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:51.061255 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.061753 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.063158 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.063379 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.063535 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.063858 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:51.063926 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:51.063984 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:51.064118 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 18:46:51.064167 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:46:51.076272 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.072764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:35031 (local address 127.29.159.67:34547)
0927 18:46:51.072959 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:51.072963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:51.072977 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:46:51.073083 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:51.073086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:51.073128 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:51.073206 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:51.073470 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.073976 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.075022 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.075250 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.075362 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.075694 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:51.075775 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:51.075991 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:51.076079 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 18:46:51.076122 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":141}
I20260927 18:46:51.076272 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.072715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:34547 (local address 127.29.159.67:35031)
0927 18:46:51.072867 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:51.072882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:51.072973 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:51.073216 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:51.073219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:51.073225 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:46:51.073369 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.073374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.074096 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.074102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.074880 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.074890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.075362 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.075366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.075487 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.075536 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:46:51.075991 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:46:51.075997 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:46:51.076062 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:46:51.076152 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":24}
I20260927 18:46:51.083549 32364 raft_consensus.cc:1261] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Refusing update from remote peer ed94647a16dd4fe8a74bce6c4fa0b58e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:51.083648 32224 raft_consensus.cc:1261] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Refusing update from remote peer ed94647a16dd4fe8a74bce6c4fa0b58e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:51.084012 32609 consensus_queue.cc:1035] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:51.084159 32583 consensus_queue.cc:1035] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:51.090034 32600 mvcc.cc:205] Tried to move back new op lower bound from 7334030586194763776 to 7334030585264369664. Current Snapshot: MvccSnapshot[applied={T|T < 7334030586194763776}]
I20260927 18:46:51.127643 32067 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:51.127821 32067 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:51.130486 32067 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9: Sending DeleteTablet for 3 replicas of tablet 0618b41f13e54cb795e70187e125cb5b
I20260927 18:46:51.131352 32204 tablet_service.cc:1514] Processing DeleteTablet for tablet 0618b41f13e54cb795e70187e125cb5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:51 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:44424
I20260927 18:46:51.131382 32344 tablet_service.cc:1514] Processing DeleteTablet for tablet 0618b41f13e54cb795e70187e125cb5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:51 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:48072
I20260927 18:46:51.131974 32615 tablet_replica.cc:330] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: stopping tablet replica
I20260927 18:46:51.132068 32067 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 18:46:51.132265 32484 tablet_service.cc:1514] Processing DeleteTablet for tablet 0618b41f13e54cb795e70187e125cb5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:51 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:39398
I20260927 18:46:51.132380 32615 raft_consensus.cc:2229] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 18:46:51.132485 32067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:51.132596 32615 raft_consensus.cc:2258] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:51.133072 32614 tablet_replica.cc:330] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: stopping tablet replica
I20260927 18:46:51.133335 32616 tablet_replica.cc:330] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: stopping tablet replica
I20260927 18:46:51.133899 32614 raft_consensus.cc:2229] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:51.134054 32616 raft_consensus.cc:2229] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:46:51.134421 32616 raft_consensus.cc:2258] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:51.134776 32614 raft_consensus.cc:2258] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:51.134914 32615 ts_tablet_manager.cc:1905] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:51.136711 32616 ts_tablet_manager.cc:1905] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:51.138139 32615 ts_tablet_manager.cc:1918] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:46:51.138242 32615 log.cc:1196] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/wal/wals/0618b41f13e54cb795e70187e125cb5b
I20260927 18:46:51.138417 32204 tablet_service.cc:1467] Processing CreateTablet for tablet 22bacacc104649f194a006c30da42fc5 (DEFAULT_TABLE table=test-table [id=9ec2a37fcf084752b23dbb0711e94716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:51.138576 32615 ts_tablet_manager.cc:1939] T 0618b41f13e54cb795e70187e125cb5b P 78b172c2925c4e99a334ff57de37e502: Deleting consensus metadata
I20260927 18:46:51.138715 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bacacc104649f194a006c30da42fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:51.139727 32616 ts_tablet_manager.cc:1918] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:46:51.139817 32616 log.cc:1196] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/0618b41f13e54cb795e70187e125cb5b
I20260927 18:46:51.139940 32051 catalog_manager.cc:4841] TS 78b172c2925c4e99a334ff57de37e502 (127.29.159.65:38795): tablet 0618b41f13e54cb795e70187e125cb5b (table test-table [id=3f7d7e2d6a7b414e8727818182dbf55b]) successfully deleted
I20260927 18:46:51.140236 32616 ts_tablet_manager.cc:1939] T 0618b41f13e54cb795e70187e125cb5b P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting consensus metadata
I20260927 18:46:51.141134 32573 tablet_bootstrap.cc:492] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502: Bootstrap starting.
I20260927 18:46:51.141481 32344 tablet_service.cc:1467] Processing CreateTablet for tablet 22bacacc104649f194a006c30da42fc5 (DEFAULT_TABLE table=test-table [id=9ec2a37fcf084752b23dbb0711e94716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:51.141594 32053 catalog_manager.cc:4841] TS ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67:34547): tablet 0618b41f13e54cb795e70187e125cb5b (table test-table [id=3f7d7e2d6a7b414e8727818182dbf55b]) successfully deleted
I20260927 18:46:51.141772 32344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bacacc104649f194a006c30da42fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:51.142074 32573 tablet_bootstrap.cc:654] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:51.143110 32573 tablet_bootstrap.cc:492] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502: No bootstrap required, opened a new log
I20260927 18:46:51.143177 32573 ts_tablet_manager.cc:1397] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:46:51.143580 32573 raft_consensus.cc:348] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.143669 32573 raft_consensus.cc:374] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:51.143693 32573 raft_consensus.cc:729] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b172c2925c4e99a334ff57de37e502, State: Initialized, Role: FOLLOWER
I20260927 18:46:51.143821 32573 consensus_queue.cc:260] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.144127 32573 ts_tablet_manager.cc:1428] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:46:51.144526 32484 tablet_service.cc:1467] Processing CreateTablet for tablet 22bacacc104649f194a006c30da42fc5 (DEFAULT_TABLE table=test-table [id=9ec2a37fcf084752b23dbb0711e94716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:51.144685 32575 tablet_bootstrap.cc:492] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6: Bootstrap starting.
I20260927 18:46:51.144776 32484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bacacc104649f194a006c30da42fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:51.145644 32575 tablet_bootstrap.cc:654] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:51.147250 32574 tablet_bootstrap.cc:492] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e: Bootstrap starting.
I20260927 18:46:51.148499 32575 tablet_bootstrap.cc:492] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6: No bootstrap required, opened a new log
I20260927 18:46:51.148597 32575 ts_tablet_manager.cc:1397] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 18:46:51.149029 32574 tablet_bootstrap.cc:654] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:51.149083 32575 raft_consensus.cc:348] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.149235 32575 raft_consensus.cc:374] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:51.149278 32575 raft_consensus.cc:729] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf86ab155a4e1c8f929bdd995431d6, State: Initialized, Role: FOLLOWER
I20260927 18:46:51.149405 32575 consensus_queue.cc:260] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.150363 32575 ts_tablet_manager.cc:1428] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 18:46:51.150676 32614 ts_tablet_manager.cc:1905] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:51.151489 32574 tablet_bootstrap.cc:492] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e: No bootstrap required, opened a new log
I20260927 18:46:51.151584 32574 ts_tablet_manager.cc:1397] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 18:46:51.152083 32574 raft_consensus.cc:348] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.152240 32574 raft_consensus.cc:374] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:51.152282 32574 raft_consensus.cc:729] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed94647a16dd4fe8a74bce6c4fa0b58e, State: Initialized, Role: FOLLOWER
I20260927 18:46:51.152406 32574 consensus_queue.cc:260] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.152884 32574 ts_tablet_manager.cc:1428] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:46:51.156163 32580 raft_consensus.cc:482] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:51.156203 32614 ts_tablet_manager.cc:1918] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:46:51.156306 32614 log.cc:1196] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/0618b41f13e54cb795e70187e125cb5b
I20260927 18:46:51.156306 32580 raft_consensus.cc:504] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.156670 32614 ts_tablet_manager.cc:1939] T 0618b41f13e54cb795e70187e125cb5b P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting consensus metadata
I20260927 18:46:51.157513 32580 leader_election.cc:288] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305), ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67:34547)
I20260927 18:46:51.159068 32053 catalog_manager.cc:4841] TS 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305): tablet 0618b41f13e54cb795e70187e125cb5b (table test-table [id=3f7d7e2d6a7b414e8727818182dbf55b]) successfully deleted
I20260927 18:46:51.163067 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.158478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:52027 (local address 127.29.159.66:41305)
0927 18:46:51.158594 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:51.158598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:51.158611 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:51.158692 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:51.158696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:51.158734 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:51.158824 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:51.159119 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.159867 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.161939 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.162456 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.162538 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.162578 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:51.162886 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 18:46:51.162955 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:46:51.163095 32621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.157703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:41305 (local address 127.29.159.65:52027)
0927 18:46:51.158475 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:51.158490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:51.158589 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:51.158837 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:51.158840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:51.158859 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:51.158992 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.159001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.160324 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.160328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.161787 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.161798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.162672 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.162674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.162755 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.162771 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 18:46:51.162858 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:46:51.162931 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260927 18:46:51.163528 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.158185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:45377 (local address 127.29.159.67:34547)
0927 18:46:51.158320 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:51.158324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:51.159225 (+ 901us) server_negotiation.cc:407] Connection header received
0927 18:46:51.159408 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:51.159411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:51.159498 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:51.159568 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:46:51.159868 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.160584 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.162078 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.162782 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.162910 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.162962 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:51.163288 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 18:46:51.163366 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:46:51.163766 32364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bacacc104649f194a006c30da42fc5" candidate_uuid: "78b172c2925c4e99a334ff57de37e502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" is_pre_election: true
I20260927 18:46:51.163900 32364 raft_consensus.cc:2454] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b172c2925c4e99a334ff57de37e502 in term 0.
I20260927 18:46:51.163902 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.158094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:34547 (local address 127.29.159.65:45377)
0927 18:46:51.159129 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:51.159144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:51.159290 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:51.159603 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:51.159607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:51.159626 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:46:51.159748 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.159755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.160720 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.160723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.161945 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.161953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.163030 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.163034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.163142 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.163164 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:46:51.163682 (+ 518us) client_negotiation.cc:240] Negotiation successful
0927 18:46:51.163765 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"thread_start_us":113,"threads_started":1}
I20260927 18:46:51.164295 32157 leader_election.cc:302] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccf86ab155a4e1c8f929bdd995431d6, 78b172c2925c4e99a334ff57de37e502; no voters:
I20260927 18:46:51.164588 32580 raft_consensus.cc:2781] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:51.164664 32580 raft_consensus.cc:482] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:51.164705 32580 raft_consensus.cc:3037] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:51.164924 32504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bacacc104649f194a006c30da42fc5" candidate_uuid: "78b172c2925c4e99a334ff57de37e502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" is_pre_election: true
I20260927 18:46:51.165148 32504 raft_consensus.cc:2454] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b172c2925c4e99a334ff57de37e502 in term 0.
I20260927 18:46:51.165980 32580 raft_consensus.cc:504] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.166348 32580 leader_election.cc:288] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [CANDIDATE]: Term 1 election: Requested vote from peers 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305), ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67:34547)
I20260927 18:46:51.166718 32364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bacacc104649f194a006c30da42fc5" candidate_uuid: "78b172c2925c4e99a334ff57de37e502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf86ab155a4e1c8f929bdd995431d6"
I20260927 18:46:51.166854 32364 raft_consensus.cc:3037] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:51.167903 32364 raft_consensus.cc:2454] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b172c2925c4e99a334ff57de37e502 in term 1.
I20260927 18:46:51.168205 32504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bacacc104649f194a006c30da42fc5" candidate_uuid: "78b172c2925c4e99a334ff57de37e502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e"
I20260927 18:46:51.168255 32157 leader_election.cc:302] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccf86ab155a4e1c8f929bdd995431d6, 78b172c2925c4e99a334ff57de37e502; no voters:
I20260927 18:46:51.168337 32504 raft_consensus.cc:3037] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:51.168416 32580 raft_consensus.cc:2781] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:51.168618 32580 raft_consensus.cc:686] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 1 LEADER]: Becoming Leader. State: Replica: 78b172c2925c4e99a334ff57de37e502, State: Running, Role: LEADER
I20260927 18:46:51.168890 32580 consensus_queue.cc:237] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:51.169551 32504 raft_consensus.cc:2454] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b172c2925c4e99a334ff57de37e502 in term 1.
I20260927 18:46:51.171312 32067 catalog_manager.cc:5495] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b172c2925c4e99a334ff57de37e502 (127.29.159.65). New cstate: current_term: 1 leader_uuid: "78b172c2925c4e99a334ff57de37e502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20260927 18:46:51.189679 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.185583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.29.159.65:38795)
0927 18:46:51.185691 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:51.185694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:51.185707 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:51.185872 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:51.185874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:51.185925 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:51.186016 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:51.186274 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.186784 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.187971 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.188226 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.188311 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.188636 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:51.189133 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:51.189296 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:51.189422 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 18:46:51.189487 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:46:51.202154 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.198217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:54893 (local address 127.29.159.67:34547)
0927 18:46:51.198358 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:51.198363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:51.198502 (+ 139us) server_negotiation.cc:407] Connection header received
0927 18:46:51.198615 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:51.198618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:51.198673 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:51.198762 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:51.199459 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.200150 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.201026 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.201314 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.201413 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.201692 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:51.201777 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:51.201854 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:51.201956 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 18:46:51.202009 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:46:51.202178 32626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.198070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:34547 (local address 127.29.159.65:54893)
0927 18:46:51.198413 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:51.198427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:51.198522 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:51.198804 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:51.198807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:51.198815 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:46:51.199294 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.199302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.200264 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.200267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.200895 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.200904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.201432 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.201435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.201511 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.201557 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:46:51.201865 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:46:51.201870 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:46:51.201924 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 18:46:51.201994 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":149,"threads_started":1}
I20260927 18:46:51.209513 32364 raft_consensus.cc:1261] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Refusing update from remote peer 78b172c2925c4e99a334ff57de37e502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:51.209722 32504 raft_consensus.cc:1261] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Refusing update from remote peer 78b172c2925c4e99a334ff57de37e502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:51.210085 32622 consensus_queue.cc:1035] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:51.210263 32580 consensus_queue.cc:1035] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:51.233372 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.229418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:38795 (local address 127.29.159.67:36495)
0927 18:46:51.229564 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:51.229581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:51.229666 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:51.230057 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:51.230060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:51.230067 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:46:51.230160 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.230165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.230943 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.230950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.231781 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:51.231794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.232451 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.232454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:51.232552 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.232582 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:46:51.233103 (+ 521us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:46:51.233110 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:46:51.233175 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:46:51.233268 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 18:46:51.233882 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:51.229627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:36495 (local address 127.29.159.65:38795)
0927 18:46:51.229739 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:51.229744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:51.229757 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:46:51.229809 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:51.229811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:51.229859 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:51.229934 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:51.230256 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.230830 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.231981 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:51.232281 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:51.232378 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:51.232745 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:51.232829 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:51.232930 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:51.233656 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 18:46:51.233723 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 18:46:51.234503 32599 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:46:52.794492 32067 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:52.794639 32067 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:52.797000 32067 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9: Sending DeleteTablet for 3 replicas of tablet 22bacacc104649f194a006c30da42fc5
I20260927 18:46:52.797559 32204 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bacacc104649f194a006c30da42fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:52 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:44424
I20260927 18:46:52.797613 32344 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bacacc104649f194a006c30da42fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:52 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:48072
I20260927 18:46:52.797750 32484 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bacacc104649f194a006c30da42fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:52 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:39398
I20260927 18:46:52.798404 32653 tablet_replica.cc:330] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502: stopping tablet replica
I20260927 18:46:52.798532 32655 tablet_replica.cc:330] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e: stopping tablet replica
I20260927 18:46:52.798547 32653 raft_consensus.cc:2229] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:46:52.798652 32655 raft_consensus.cc:2229] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:52.798763 32655 raft_consensus.cc:2258] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:52.798866 32653 raft_consensus.cc:2258] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:52.799114 32654 tablet_replica.cc:330] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6: stopping tablet replica
I20260927 18:46:52.799247 32654 raft_consensus.cc:2229] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:52.799355 32654 raft_consensus.cc:2258] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:52.799458 32655 ts_tablet_manager.cc:1905] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:52.800161 32654 ts_tablet_manager.cc:1905] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:52.801704 32653 ts_tablet_manager.cc:1905] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:52.802482 32654 ts_tablet_manager.cc:1918] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:46:52.802579 32654 log.cc:1196] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/22bacacc104649f194a006c30da42fc5
I20260927 18:46:52.802635 32655 ts_tablet_manager.cc:1918] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:46:52.802718 32655 log.cc:1196] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/22bacacc104649f194a006c30da42fc5
I20260927 18:46:52.802881 32654 ts_tablet_manager.cc:1939] T 22bacacc104649f194a006c30da42fc5 P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting consensus metadata
I20260927 18:46:52.802989 32655 ts_tablet_manager.cc:1939] T 22bacacc104649f194a006c30da42fc5 P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting consensus metadata
I20260927 18:46:52.803813 32053 catalog_manager.cc:4841] TS ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67:34547): tablet 22bacacc104649f194a006c30da42fc5 (table test-table [id=9ec2a37fcf084752b23dbb0711e94716]) successfully deleted
I20260927 18:46:52.804538 32653 ts_tablet_manager.cc:1918] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:46:52.804618 32653 log.cc:1196] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/wal/wals/22bacacc104649f194a006c30da42fc5
I20260927 18:46:52.804744 32053 catalog_manager.cc:4841] TS 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305): tablet 22bacacc104649f194a006c30da42fc5 (table test-table [id=9ec2a37fcf084752b23dbb0711e94716]) successfully deleted
I20260927 18:46:52.804931 32653 ts_tablet_manager.cc:1939] T 22bacacc104649f194a006c30da42fc5 P 78b172c2925c4e99a334ff57de37e502: Deleting consensus metadata
I20260927 18:46:52.805799 32051 catalog_manager.cc:4841] TS 78b172c2925c4e99a334ff57de37e502 (127.29.159.65:38795): tablet 22bacacc104649f194a006c30da42fc5 (table test-table [id=9ec2a37fcf084752b23dbb0711e94716]) successfully deleted
Sep 27 18:46:52 dist-test-slave-8tg2 krb5kdc[32022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:46:52 dist-test-slave-8tg2 krb5kdc[32022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:52.828145 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:52.819263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.29.159.126:46685)
0927 18:46:52.819613 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:52.819628 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:52.819745 (+ 117us) server_negotiation.cc:407] Connection header received
0927 18:46:52.819818 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:52.819821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:52.819871 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:52.819967 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:52.820770 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:52.821384 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:52.822232 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:52.822460 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:52.822549 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:52.825142 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:52.825145 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:52.825147 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:52.825172 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:52.826574 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:52.826953 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:52.826955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:52.826956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:52.827030 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:52.827258 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:52.827260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:52.827262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:52.827496 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:52.827602 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:52.827838 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 18:46:52.827939 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":77,"threads_started":1}
I20260927 18:46:52.830897 32067 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:46:52.831298 32067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:52.835815 32344 tablet_service.cc:1467] Processing CreateTablet for tablet aa472a95e4794ba0ad950dd12da2baa7 (DEFAULT_TABLE table=test-table [id=f036637df63348099535846d2827e0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:52.835844 32484 tablet_service.cc:1467] Processing CreateTablet for tablet aa472a95e4794ba0ad950dd12da2baa7 (DEFAULT_TABLE table=test-table [id=f036637df63348099535846d2827e0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:52.835871 32204 tablet_service.cc:1467] Processing CreateTablet for tablet aa472a95e4794ba0ad950dd12da2baa7 (DEFAULT_TABLE table=test-table [id=f036637df63348099535846d2827e0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:52.836105 32344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa472a95e4794ba0ad950dd12da2baa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:52.836105 32484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa472a95e4794ba0ad950dd12da2baa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:52.836170 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa472a95e4794ba0ad950dd12da2baa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:52.838610 32665 tablet_bootstrap.cc:492] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e: Bootstrap starting.
I20260927 18:46:52.838627 32664 tablet_bootstrap.cc:492] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502: Bootstrap starting.
I20260927 18:46:52.838950 32666 tablet_bootstrap.cc:492] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6: Bootstrap starting.
I20260927 18:46:52.839522 32665 tablet_bootstrap.cc:654] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:52.839546 32664 tablet_bootstrap.cc:654] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:52.839809 32666 tablet_bootstrap.cc:654] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:52.840641 32664 tablet_bootstrap.cc:492] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502: No bootstrap required, opened a new log
I20260927 18:46:52.840641 32665 tablet_bootstrap.cc:492] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e: No bootstrap required, opened a new log
I20260927 18:46:52.840728 32664 ts_tablet_manager.cc:1397] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:46:52.840729 32665 ts_tablet_manager.cc:1397] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:46:52.841024 32666 tablet_bootstrap.cc:492] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6: No bootstrap required, opened a new log
I20260927 18:46:52.841109 32666 ts_tablet_manager.cc:1397] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:52.841114 32665 raft_consensus.cc:348] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:52.841114 32664 raft_consensus.cc:348] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:52.841219 32665 raft_consensus.cc:374] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:52.841219 32664 raft_consensus.cc:374] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:52.841254 32664 raft_consensus.cc:729] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b172c2925c4e99a334ff57de37e502, State: Initialized, Role: FOLLOWER
I20260927 18:46:52.841254 32665 raft_consensus.cc:729] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed94647a16dd4fe8a74bce6c4fa0b58e, State: Initialized, Role: FOLLOWER
I20260927 18:46:52.841405 32665 consensus_queue.cc:260] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:52.841406 32664 consensus_queue.cc:260] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:52.841748 32665 ts_tablet_manager.cc:1428] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:46:52.841535 32666 raft_consensus.cc:348] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:52.842025 32666 raft_consensus.cc:374] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:52.842077 32666 raft_consensus.cc:729] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf86ab155a4e1c8f929bdd995431d6, State: Initialized, Role: FOLLOWER
I20260927 18:46:52.842118 32664 ts_tablet_manager.cc:1428] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:46:52.842218 32666 consensus_queue.cc:260] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:52.842561 32666 ts_tablet_manager.cc:1428] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:46:52.967962 32634 raft_consensus.cc:482] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:52.968106 32634 raft_consensus.cc:504] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:52.968590 32634 leader_election.cc:288] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b172c2925c4e99a334ff57de37e502 (127.29.159.65:38795), 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305)
I20260927 18:46:52.969079 32364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa472a95e4794ba0ad950dd12da2baa7" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" is_pre_election: true
I20260927 18:46:52.969079 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa472a95e4794ba0ad950dd12da2baa7" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b172c2925c4e99a334ff57de37e502" is_pre_election: true
I20260927 18:46:52.969239 32224 raft_consensus.cc:2454] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 0.
I20260927 18:46:52.969239 32364 raft_consensus.cc:2454] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 0.
I20260927 18:46:52.969604 32437 leader_election.cc:302] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ccf86ab155a4e1c8f929bdd995431d6, 78b172c2925c4e99a334ff57de37e502, ed94647a16dd4fe8a74bce6c4fa0b58e; no voters:
I20260927 18:46:52.969883 32634 raft_consensus.cc:2781] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:52.969977 32634 raft_consensus.cc:482] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:52.970034 32634 raft_consensus.cc:3037] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:52.971216 32634 raft_consensus.cc:504] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:52.971621 32634 leader_election.cc:288] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 election: Requested vote from peers 78b172c2925c4e99a334ff57de37e502 (127.29.159.65:38795), 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305)
I20260927 18:46:52.972031 32364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa472a95e4794ba0ad950dd12da2baa7" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf86ab155a4e1c8f929bdd995431d6"
I20260927 18:46:52.972031 32224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa472a95e4794ba0ad950dd12da2baa7" candidate_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b172c2925c4e99a334ff57de37e502"
I20260927 18:46:52.972162 32364 raft_consensus.cc:3037] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:52.972257 32224 raft_consensus.cc:3037] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:52.973385 32364 raft_consensus.cc:2454] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 1.
I20260927 18:46:52.973695 32437 leader_election.cc:302] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccf86ab155a4e1c8f929bdd995431d6, ed94647a16dd4fe8a74bce6c4fa0b58e; no voters:
I20260927 18:46:52.973690 32224 raft_consensus.cc:2454] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed94647a16dd4fe8a74bce6c4fa0b58e in term 1.
I20260927 18:46:52.973896 32634 raft_consensus.cc:2781] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:52.974018 32634 raft_consensus.cc:686] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 LEADER]: Becoming Leader. State: Replica: ed94647a16dd4fe8a74bce6c4fa0b58e, State: Running, Role: LEADER
I20260927 18:46:52.974162 32634 consensus_queue.cc:237] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:52.975862 32066 catalog_manager.cc:5495] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e reported cstate change: term changed from 0 to 1, leader changed from <none> to ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67). New cstate: current_term: 1 leader_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } health_report { overall_health: HEALTHY } } }
I20260927 18:46:52.999693 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:52.996413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.29.159.67:34547)
0927 18:46:52.996720 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:52.996730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:52.996745 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:46:52.996784 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:52.996787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:52.996834 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:52.996921 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:52.997172 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:52.997697 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:52.998580 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:52.998827 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:52.998937 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:52.999195 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:52.999274 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:52.999355 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:52.999483 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 18:46:52.999535 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":128,"threads_started":1}
I20260927 18:46:53.006199 32364 raft_consensus.cc:1261] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Refusing update from remote peer ed94647a16dd4fe8a74bce6c4fa0b58e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:53.006207 32224 raft_consensus.cc:1261] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Refusing update from remote peer ed94647a16dd4fe8a74bce6c4fa0b58e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:53.006608 32639 consensus_queue.cc:1035] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:53.006767 32634 consensus_queue.cc:1035] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:53.035892 32066 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:53.036031 32066 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:53.037942 32066 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9: Sending DeleteTablet for 3 replicas of tablet aa472a95e4794ba0ad950dd12da2baa7
I20260927 18:46:53.038498 32204 tablet_service.cc:1514] Processing DeleteTablet for tablet aa472a95e4794ba0ad950dd12da2baa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:53 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:44424
I20260927 18:46:53.038579 32344 tablet_service.cc:1514] Processing DeleteTablet for tablet aa472a95e4794ba0ad950dd12da2baa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:53 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:48072
I20260927 18:46:53.038604 32483 tablet_service.cc:1514] Processing DeleteTablet for tablet aa472a95e4794ba0ad950dd12da2baa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:53 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:39398
I20260927 18:46:53.038692 32653 tablet_replica.cc:330] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502: stopping tablet replica
I20260927 18:46:53.038777 32655 tablet_replica.cc:330] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e: stopping tablet replica
I20260927 18:46:53.038800 32654 tablet_replica.cc:330] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6: stopping tablet replica
I20260927 18:46:53.038822 32653 raft_consensus.cc:2229] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:53.038897 32655 raft_consensus.cc:2229] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:46:53.038928 32654 raft_consensus.cc:2229] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:53.038942 32653 raft_consensus.cc:2258] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:53.039043 32654 raft_consensus.cc:2258] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:53.039076 32655 raft_consensus.cc:2258] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:53.039861 32654 ts_tablet_manager.cc:1905] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:53.040012 32653 ts_tablet_manager.cc:1905] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:53.040051 32655 ts_tablet_manager.cc:1905] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:53.042114 32655 ts_tablet_manager.cc:1918] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:46:53.042212 32655 log.cc:1196] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/aa472a95e4794ba0ad950dd12da2baa7
I20260927 18:46:53.042379 32653 ts_tablet_manager.cc:1918] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:46:53.042379 32654 ts_tablet_manager.cc:1918] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:46:53.042460 32654 log.cc:1196] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/aa472a95e4794ba0ad950dd12da2baa7
I20260927 18:46:53.042460 32653 log.cc:1196] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-0/wal/wals/aa472a95e4794ba0ad950dd12da2baa7
I20260927 18:46:53.042492 32655 ts_tablet_manager.cc:1939] T aa472a95e4794ba0ad950dd12da2baa7 P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting consensus metadata
I20260927 18:46:53.042729 32653 ts_tablet_manager.cc:1939] T aa472a95e4794ba0ad950dd12da2baa7 P 78b172c2925c4e99a334ff57de37e502: Deleting consensus metadata
I20260927 18:46:53.042733 32654 ts_tablet_manager.cc:1939] T aa472a95e4794ba0ad950dd12da2baa7 P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting consensus metadata
I20260927 18:46:53.042958 32066 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 18:46:53.043352 32053 catalog_manager.cc:4841] TS ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67:34547): tablet aa472a95e4794ba0ad950dd12da2baa7 (table test-table [id=f036637df63348099535846d2827e0d9]) successfully deleted
W20260927 18:46:53.043483 32066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:46:53.043762 32051 catalog_manager.cc:4841] TS 78b172c2925c4e99a334ff57de37e502 (127.29.159.65:38795): tablet aa472a95e4794ba0ad950dd12da2baa7 (table test-table [id=f036637df63348099535846d2827e0d9]) successfully deleted
I20260927 18:46:53.044198 32053 catalog_manager.cc:4841] TS 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305): tablet aa472a95e4794ba0ad950dd12da2baa7 (table test-table [id=f036637df63348099535846d2827e0d9]) successfully deleted
I20260927 18:46:53.048096 32344 tablet_service.cc:1467] Processing CreateTablet for tablet b49194465d7b4b4fa05b3d26fb4ac8dc (DEFAULT_TABLE table=test-table [id=dd6744e1aace496485825c1b8fd7ef60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:53.048107 32204 tablet_service.cc:1467] Processing CreateTablet for tablet b49194465d7b4b4fa05b3d26fb4ac8dc (DEFAULT_TABLE table=test-table [id=dd6744e1aace496485825c1b8fd7ef60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:53.048369 32344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49194465d7b4b4fa05b3d26fb4ac8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:53.048369 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49194465d7b4b4fa05b3d26fb4ac8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:53.048385 32483 tablet_service.cc:1467] Processing CreateTablet for tablet b49194465d7b4b4fa05b3d26fb4ac8dc (DEFAULT_TABLE table=test-table [id=dd6744e1aace496485825c1b8fd7ef60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:46:53.048614 32483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49194465d7b4b4fa05b3d26fb4ac8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:53.050536 32666 tablet_bootstrap.cc:492] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6: Bootstrap starting.
I20260927 18:46:53.050611 32665 tablet_bootstrap.cc:492] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e: Bootstrap starting.
I20260927 18:46:53.051412 32665 tablet_bootstrap.cc:654] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:53.051581 32666 tablet_bootstrap.cc:654] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:53.052232 32664 tablet_bootstrap.cc:492] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502: Bootstrap starting.
I20260927 18:46:53.052716 32665 tablet_bootstrap.cc:492] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e: No bootstrap required, opened a new log
I20260927 18:46:53.052783 32665 ts_tablet_manager.cc:1397] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:46:53.052865 32666 tablet_bootstrap.cc:492] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6: No bootstrap required, opened a new log
I20260927 18:46:53.052943 32666 ts_tablet_manager.cc:1397] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:46:53.053076 32664 tablet_bootstrap.cc:654] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:53.053200 32665 raft_consensus.cc:348] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:53.053294 32665 raft_consensus.cc:374] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:53.053316 32665 raft_consensus.cc:729] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed94647a16dd4fe8a74bce6c4fa0b58e, State: Initialized, Role: FOLLOWER
I20260927 18:46:53.053438 32665 consensus_queue.cc:260] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:53.053399 32666 raft_consensus.cc:348] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:53.053532 32666 raft_consensus.cc:374] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:53.053570 32666 raft_consensus.cc:729] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf86ab155a4e1c8f929bdd995431d6, State: Initialized, Role: FOLLOWER
I20260927 18:46:53.053730 32665 ts_tablet_manager.cc:1428] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:46:53.053722 32666 consensus_queue.cc:260] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:53.053973 32666 ts_tablet_manager.cc:1428] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:46:53.055269 32664 tablet_bootstrap.cc:492] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502: No bootstrap required, opened a new log
I20260927 18:46:53.055361 32664 ts_tablet_manager.cc:1397] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 18:46:53.055769 32664 raft_consensus.cc:348] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:53.055909 32664 raft_consensus.cc:374] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:53.055950 32664 raft_consensus.cc:729] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b172c2925c4e99a334ff57de37e502, State: Initialized, Role: FOLLOWER
I20260927 18:46:53.056087 32664 consensus_queue.cc:260] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:53.056347 32664 ts_tablet_manager.cc:1428] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:46:53.078454 32644 raft_consensus.cc:482] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:46:53.078606 32644 raft_consensus.cc:504] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:53.079057 32644 leader_election.cc:288] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305), ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67:34547)
I20260927 18:46:53.079536 32504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49194465d7b4b4fa05b3d26fb4ac8dc" candidate_uuid: "78b172c2925c4e99a334ff57de37e502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" is_pre_election: true
I20260927 18:46:53.079533 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49194465d7b4b4fa05b3d26fb4ac8dc" candidate_uuid: "78b172c2925c4e99a334ff57de37e502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" is_pre_election: true
I20260927 18:46:53.079700 32504 raft_consensus.cc:2454] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b172c2925c4e99a334ff57de37e502 in term 0.
I20260927 18:46:53.079700 32363 raft_consensus.cc:2454] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b172c2925c4e99a334ff57de37e502 in term 0.
I20260927 18:46:53.080044 32157 leader_election.cc:302] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b172c2925c4e99a334ff57de37e502, ed94647a16dd4fe8a74bce6c4fa0b58e; no voters:
I20260927 18:46:53.080250 32644 raft_consensus.cc:2781] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:46:53.080310 32644 raft_consensus.cc:482] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:46:53.080363 32644 raft_consensus.cc:3037] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:53.081418 32644 raft_consensus.cc:504] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:53.081738 32644 leader_election.cc:288] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [CANDIDATE]: Term 1 election: Requested vote from peers 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305), ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67:34547)
I20260927 18:46:53.082057 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49194465d7b4b4fa05b3d26fb4ac8dc" candidate_uuid: "78b172c2925c4e99a334ff57de37e502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccf86ab155a4e1c8f929bdd995431d6"
I20260927 18:46:53.082158 32504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49194465d7b4b4fa05b3d26fb4ac8dc" candidate_uuid: "78b172c2925c4e99a334ff57de37e502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e"
I20260927 18:46:53.082216 32363 raft_consensus.cc:3037] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:53.082305 32504 raft_consensus.cc:3037] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:53.083225 32504 raft_consensus.cc:2454] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b172c2925c4e99a334ff57de37e502 in term 1.
I20260927 18:46:53.083472 32363 raft_consensus.cc:2454] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b172c2925c4e99a334ff57de37e502 in term 1.
I20260927 18:46:53.083539 32157 leader_election.cc:302] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b172c2925c4e99a334ff57de37e502, ed94647a16dd4fe8a74bce6c4fa0b58e; no voters:
I20260927 18:46:53.083706 32644 raft_consensus.cc:2781] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:53.083806 32644 raft_consensus.cc:686] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [term 1 LEADER]: Becoming Leader. State: Replica: 78b172c2925c4e99a334ff57de37e502, State: Running, Role: LEADER
I20260927 18:46:53.083967 32644 consensus_queue.cc:237] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } }
I20260927 18:46:53.085623 32067 catalog_manager.cc:5495] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b172c2925c4e99a334ff57de37e502 (127.29.159.65). New cstate: current_term: 1 leader_uuid: "78b172c2925c4e99a334ff57de37e502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b172c2925c4e99a334ff57de37e502" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 } health_report { overall_health: UNKNOWN } } }
I20260927 18:46:53.105551 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:53.102232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.29.159.65:38795)
0927 18:46:53.102572 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:53.102580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:53.102592 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:46:53.102628 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:53.102630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:53.102670 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:53.102775 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:46:53.103042 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:53.103593 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:53.104439 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:53.104670 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:53.104789 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:53.105065 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:46:53.105140 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:46:53.105223 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:53.105325 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 18:46:53.105384 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":129,"threads_started":1}
I20260927 18:46:53.112309 32363 raft_consensus.cc:1261] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Refusing update from remote peer 78b172c2925c4e99a334ff57de37e502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:53.112315 32504 raft_consensus.cc:1261] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Refusing update from remote peer 78b172c2925c4e99a334ff57de37e502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:46:53.112720 32644 consensus_queue.cc:1035] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccf86ab155a4e1c8f929bdd995431d6" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:46:53.112874 32580 consensus_queue.cc:1035] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed94647a16dd4fe8a74bce6c4fa0b58e" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:46:53.126096 32635 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:46:54.679010 32066 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:54.679164 32066 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:46:54.681567 32066 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b934b2797c6841bc984b6115acfd8ec9: Sending DeleteTablet for 3 replicas of tablet b49194465d7b4b4fa05b3d26fb4ac8dc
I20260927 18:46:54.682138 32204 tablet_service.cc:1514] Processing DeleteTablet for tablet b49194465d7b4b4fa05b3d26fb4ac8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:54 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:44424
I20260927 18:46:54.682216 32344 tablet_service.cc:1514] Processing DeleteTablet for tablet b49194465d7b4b4fa05b3d26fb4ac8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:54 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:48072
I20260927 18:46:54.682279 32483 tablet_service.cc:1514] Processing DeleteTablet for tablet b49194465d7b4b4fa05b3d26fb4ac8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:46:54 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:39398
I20260927 18:46:54.682641 32706 tablet_replica.cc:330] T b49194465d7b4b4fa05b3d26fb4ac8dc P 78b172c2925c4e99a334ff57de37e502: stopping tablet replica
I20260927 18:46:54.682643 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 32139
I20260927 18:46:54.682648 32708 tablet_replica.cc:330] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e: stopping tablet replica
I20260927 18:46:54.684065 32707 tablet_replica.cc:330] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6: stopping tablet replica
I20260927 18:46:54.684248 32707 raft_consensus.cc:2229] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:54.684393 32707 raft_consensus.cc:2258] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:54.684527 32708 raft_consensus.cc:2229] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:46:54.684684 32708 raft_consensus.cc:2258] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:46:54.685216 32707 ts_tablet_manager.cc:1905] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:54.685364 32708 ts_tablet_manager.cc:1905] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:46:54.687491 32708 ts_tablet_manager.cc:1918] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:46:54.687609 32708 log.cc:1196] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/b49194465d7b4b4fa05b3d26fb4ac8dc
I20260927 18:46:54.687883 32708 ts_tablet_manager.cc:1939] T b49194465d7b4b4fa05b3d26fb4ac8dc P ed94647a16dd4fe8a74bce6c4fa0b58e: Deleting consensus metadata
I20260927 18:46:54.688719 32053 catalog_manager.cc:4841] TS ed94647a16dd4fe8a74bce6c4fa0b58e (127.29.159.67:34547): tablet b49194465d7b4b4fa05b3d26fb4ac8dc (table test-table [id=dd6744e1aace496485825c1b8fd7ef60]) successfully deleted
I20260927 18:46:54.688743 32707 ts_tablet_manager.cc:1918] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:46:54.688829 32707 log.cc:1196] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/b49194465d7b4b4fa05b3d26fb4ac8dc
I20260927 18:46:54.689100 32707 ts_tablet_manager.cc:1939] T b49194465d7b4b4fa05b3d26fb4ac8dc P 3ccf86ab155a4e1c8f929bdd995431d6: Deleting consensus metadata
I20260927 18:46:54.689908 32053 catalog_manager.cc:4841] TS 3ccf86ab155a4e1c8f929bdd995431d6 (127.29.159.66:41305): tablet b49194465d7b4b4fa05b3d26fb4ac8dc (table test-table [id=dd6744e1aace496485825c1b8fd7ef60]) successfully deleted
W20260927 18:46:54.691124 32051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:38795 (error 108)
I20260927 18:46:54.691453 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 32279
I20260927 18:46:54.692703 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:54.692218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:38795 (local address 127.0.0.1:44146)
0927 18:46:54.692518 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:54.692600 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:38795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
W20260927 18:46:54.692859 32051 catalog_manager.cc:4584] TS 78b172c2925c4e99a334ff57de37e502 (127.29.159.65:38795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b49194465d7b4b4fa05b3d26fb4ac8dc: Network error: Client connection negotiation failed: client connection to 127.29.159.65:38795: connect: Connection refused (error 111)
I20260927 18:46:54.698402 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 32419
I20260927 18:46:54.705044 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 32035
2026-09-27T18:46:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6122 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:54 dist-test-slave-8tg2 krb5kdc[32713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:54 dist-test-slave-8tg2 krb5kdc[32713](info): set up 2 sockets
Sep 27 18:46:54 dist-test-slave-8tg2 krb5kdc[32713](info): commencing operation
krb5kdc: starting...
W20260927 18:46:55.735504 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:46:55 dist-test-slave-8tg2 krb5kdc[32713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:46:55.855021 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:43285
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:41463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:43285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:55.965343 32726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:55.965649 32726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:55.965692 32726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:55.969851 32726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:55.969920 32726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:55.969940 32726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:55.969956 32726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:55.969972 32726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:55.975541 32726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:41463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:43285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:43285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:55.976780 32726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:55.977912 32726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:55.986905 32731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:55.986948 32734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:55.987026 32732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:55.987068 32726 server_base.cc:1034] running on GCE node
I20260927 18:46:55.987819 32726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:55.988900 32726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:55.990108 32726 hybrid_clock.cc:648] HybridClock initialized: now 1790534815990079 us; error 33 us; skew 500 ppm
Sep 27 18:46:55 dist-test-slave-8tg2 krb5kdc[32713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:55.999312 32726 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:46:56.000648 32726 webserver.cc:460] Webserver started at http://127.29.159.126:34909/ using document root <none> and password file <none>
I20260927 18:46:56.001304 32726 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:56.001372 32726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:56.001565 32726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:56.003839 32726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "c3671a5dfada410699413713958f1d90"
format_stamp: "Formatted at 2026-09-27 18:46:55 on dist-test-slave-8tg2"
server_key: "11ac615d337b16cb4967f982a033d23f"
server_key_iv: "1b2c4d5a097269e58f9ad2e4638ea2fa"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.004383 32726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "c3671a5dfada410699413713958f1d90"
format_stamp: "Formatted at 2026-09-27 18:46:55 on dist-test-slave-8tg2"
server_key: "11ac615d337b16cb4967f982a033d23f"
server_key_iv: "1b2c4d5a097269e58f9ad2e4638ea2fa"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.008353 32726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:46:56.010933 32741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:56.012270 32726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:46:56.012391 32726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "c3671a5dfada410699413713958f1d90"
format_stamp: "Formatted at 2026-09-27 18:46:55 on dist-test-slave-8tg2"
server_key: "11ac615d337b16cb4967f982a033d23f"
server_key_iv: "1b2c4d5a097269e58f9ad2e4638ea2fa"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.012490 32726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:56.029510 32726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:56.032708 32726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:56.032874 32726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:56.040189 32726 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:43285
I20260927 18:46:56.040208 325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:43285 every 8 connection(s)
I20260927 18:46:56.041191 32726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:56.044034 326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:56.049492 326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90: Bootstrap starting.
I20260927 18:46:56.051560 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 32726
I20260927 18:46:56.051720 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:46:56.051987 30333 external_mini_cluster.cc:1428] Setting key 3b864b7719513ce1634dd3a88a19f815
I20260927 18:46:56.052026 326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:56.052892 326 log.cc:824] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:56.054766 326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90: No bootstrap required, opened a new log
I20260927 18:46:56.058902 326 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3671a5dfada410699413713958f1d90" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43285 } }
I20260927 18:46:56.059192 326 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:56.059259 326 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3671a5dfada410699413713958f1d90, State: Initialized, Role: FOLLOWER
I20260927 18:46:56.059770 326 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3671a5dfada410699413713958f1d90" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43285 } }
I20260927 18:46:56.059940 326 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:56.060037 326 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:56.060161 326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:56.061125 326 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3671a5dfada410699413713958f1d90" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43285 } }
Sep 27 18:46:56 dist-test-slave-8tg2 krb5kdc[32713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:56.061517 326 leader_election.cc:302] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3671a5dfada410699413713958f1d90; no voters:
I20260927 18:46:56.061895 326 leader_election.cc:288] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:56.061966 331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:56.062194 331 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [term 1 LEADER]: Becoming Leader. State: Replica: c3671a5dfada410699413713958f1d90, State: Running, Role: LEADER
I20260927 18:46:56.062487 331 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3671a5dfada410699413713958f1d90" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43285 } }
I20260927 18:46:56.063248 326 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:56.064391 333 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3671a5dfada410699413713958f1d90. Latest consensus state: current_term: 1 leader_uuid: "c3671a5dfada410699413713958f1d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3671a5dfada410699413713958f1d90" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43285 } } }
I20260927 18:46:56.064523 333 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:56.064802 332 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3671a5dfada410699413713958f1d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3671a5dfada410699413713958f1d90" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43285 } } }
I20260927 18:46:56.064909 332 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:56.065066 340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:56.068377 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:56.053240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.29.159.126:43285)
0927 18:46:56.053705 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:56.053721 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:56.053757 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:46:56.054382 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:56.054398 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:56.054732 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:56.055058 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:56.055983 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.057073 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.058382 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.058910 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.059046 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:56.062158 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:56.062164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:56.062178 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:56.062235 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:56.064576 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:56.065667 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:56.065675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:56.065680 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:56.065815 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:56.066126 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:56.066129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:56.066130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:56.066629 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:56.066832 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:56.067118 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 18:46:56.067417 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":159,"threads_started":1}
I20260927 18:46:56.068432 340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:56.074005 340 catalog_manager.cc:1347] Generated new cluster ID: 57a6fd33f44843518c3d95ed28a86b5e
I20260927 18:46:56.074079 340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:56.080641 340 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:56.081605 340 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:56.089882 340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3671a5dfada410699413713958f1d90: Generated new TSK 0
I20260927 18:46:56.090569 340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:46:56.122298 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43285
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:41463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:56.260492 352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:56.260746 352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:56.260787 352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:56.264756 352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:56.264806 352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:56.264861 352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:46:56.269934 352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:41463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:56.270854 352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:56.271833 352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:56.280978 358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:56.281042 352 server_base.cc:1034] running on GCE node
W20260927 18:46:56.281025 360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:56.280999 357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:56.281582 352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:56.282171 352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:56.283335 352 hybrid_clock.cc:648] HybridClock initialized: now 1790534816283302 us; error 44 us; skew 500 ppm
Sep 27 18:46:56 dist-test-slave-8tg2 krb5kdc[32713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534816, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:56.286155 352 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:46:56.287030 352 webserver.cc:460] Webserver started at http://127.29.159.65:40199/ using document root <none> and password file <none>
I20260927 18:46:56.287669 352 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:56.287747 352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:56.287979 352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:56.290055 352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "8ba7b90d85824a688533bb5ff96eb7ff"
format_stamp: "Formatted at 2026-09-27 18:46:56 on dist-test-slave-8tg2"
server_key: "77cfce49d88d9c34977b2b0d5253b2bd"
server_key_iv: "e9c325b16cff1b60532121e6d76f01e8"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.290622 352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "8ba7b90d85824a688533bb5ff96eb7ff"
format_stamp: "Formatted at 2026-09-27 18:46:56 on dist-test-slave-8tg2"
server_key: "77cfce49d88d9c34977b2b0d5253b2bd"
server_key_iv: "e9c325b16cff1b60532121e6d76f01e8"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.294437 352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:46:56.296952 367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:56.298141 352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:46:56.298310 352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "8ba7b90d85824a688533bb5ff96eb7ff"
format_stamp: "Formatted at 2026-09-27 18:46:56 on dist-test-slave-8tg2"
server_key: "77cfce49d88d9c34977b2b0d5253b2bd"
server_key_iv: "e9c325b16cff1b60532121e6d76f01e8"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.298422 352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:56.319918 352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:56.322854 352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:56.323063 352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:56.323700 352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:56.324548 352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:56.324620 352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:56.324690 352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:56.324743 352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:56.334695 352 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:36195
I20260927 18:46:56.334713 480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:36195 every 8 connection(s)
I20260927 18:46:56.335628 352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:56.339669 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 352
I20260927 18:46:56.339759 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:56.340051 30333 external_mini_cluster.cc:1428] Setting key 5de5e463f2a7b61ebd51012778799897
Sep 27 18:46:56 dist-test-slave-8tg2 krb5kdc[32713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534816, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:46:56.350898 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:56.337396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:46489 (local address 127.29.159.126:43285)
0927 18:46:56.337535 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:56.337538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:56.338424 (+ 886us) server_negotiation.cc:407] Connection header received
0927 18:46:56.340193 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:56.340197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:56.340253 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:56.340350 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:56.341815 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.342693 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.343920 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.344158 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.344266 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:56.346943 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:56.346947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:56.346950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:56.346982 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:56.348700 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:56.349189 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:56.349194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:56.349198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:56.349325 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:56.349646 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:56.349649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:56.349651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:56.349875 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:56.350000 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:56.350487 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 18:46:56.350648 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:46:56.351640 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:56.337725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:43285 (local address 127.29.159.65:46489)
0927 18:46:56.338154 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:56.338243 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:56.339962 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:56.340549 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:56.340561 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:56.341036 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:56.341661 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:56.341676 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.342818 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.342821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:56.343800 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:56.343808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.344326 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.344331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:56.344496 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:56.345096 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:56.345112 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:56.346768 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:56.348820 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:56.348825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:56.348837 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:56.349086 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:56.349427 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:56.349429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:56.349430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:56.349554 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:56.350028 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:56.350033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:56.350254 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 18:46:56.350423 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 18:46:56.350664 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
I20260927 18:46:56.352998 481 heartbeater.cc:347] Connected to a master server at 127.29.159.126:43285
I20260927 18:46:56.353248 481 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:56.353852 481 heartbeater.cc:511] Master 127.29.159.126:43285 requested a full tablet report, sending...
I20260927 18:46:56.355470 32758 ts_manager.cc:194] Registered new tserver with Master: 8ba7b90d85824a688533bb5ff96eb7ff (127.29.159.65:36195)
I20260927 18:46:56.356596 32758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:46489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:46:56.366207 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43285
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:41463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:56.475679 486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:56.475935 486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:56.475977 486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:56.480039 486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:56.480100 486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:56.480159 486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:46:56.485446 486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:41463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:56.486497 486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:56.487615 486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:56.497340 491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:56.497344 494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:56.497344 492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:56.497695 486 server_base.cc:1034] running on GCE node
I20260927 18:46:56.498162 486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:56.498787 486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:56.499935 486 hybrid_clock.cc:648] HybridClock initialized: now 1790534816499901 us; error 46 us; skew 500 ppm
Sep 27 18:46:56 dist-test-slave-8tg2 krb5kdc[32713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534816, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:56.502768 486 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:46:56.503892 486 webserver.cc:460] Webserver started at http://127.29.159.66:43987/ using document root <none> and password file <none>
I20260927 18:46:56.504460 486 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:56.504522 486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:56.504731 486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:56.506742 486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "f0be148b6f4344cba5f74aedc2510ccb"
format_stamp: "Formatted at 2026-09-27 18:46:56 on dist-test-slave-8tg2"
server_key: "521716763b5dc85a704105531593cfc1"
server_key_iv: "b7906007b24c2deb41d8de94e81c9ea1"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.507313 486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "f0be148b6f4344cba5f74aedc2510ccb"
format_stamp: "Formatted at 2026-09-27 18:46:56 on dist-test-slave-8tg2"
server_key: "521716763b5dc85a704105531593cfc1"
server_key_iv: "b7906007b24c2deb41d8de94e81c9ea1"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.511010 486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:46:56.513523 501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:56.514720 486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:56.514818 486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "f0be148b6f4344cba5f74aedc2510ccb"
format_stamp: "Formatted at 2026-09-27 18:46:56 on dist-test-slave-8tg2"
server_key: "521716763b5dc85a704105531593cfc1"
server_key_iv: "b7906007b24c2deb41d8de94e81c9ea1"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.514948 486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:56.535089 486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:56.538189 486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:56.538408 486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:56.539022 486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:56.539882 486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:56.539953 486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:56.540019 486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:56.540061 486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:56.548866 486 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:36635
I20260927 18:46:56.548913 614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:36635 every 8 connection(s)
I20260927 18:46:56.549774 486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:46:56.552307 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 486
I20260927 18:46:56.552423 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:46:56.552726 30333 external_mini_cluster.cc:1428] Setting key 783d3c5c1177e2705a6b2f793fb9e5eb
Sep 27 18:46:56 dist-test-slave-8tg2 krb5kdc[32713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534816, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:46:56.564186 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:56.551573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:35601 (local address 127.29.159.126:43285)
0927 18:46:56.551708 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:56.551712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:56.552679 (+ 967us) server_negotiation.cc:407] Connection header received
0927 18:46:56.554223 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:56.554227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:56.554277 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:56.554352 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:56.555541 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.556071 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.557211 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.557449 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.557561 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:56.560471 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:56.560474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:56.560476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:56.560500 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:56.561907 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:56.562528 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:56.562531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:56.562532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:56.562603 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:56.562941 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:56.562943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:56.562944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:56.563111 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:56.563182 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:56.563844 (+ 662us) server_negotiation.cc:299] Negotiation successful
0927 18:46:56.563946 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:46:56.565323 617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:56.551961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:43285 (local address 127.29.159.66:35601)
0927 18:46:56.552532 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:56.552574 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:56.554000 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:56.554522 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:56.554530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:56.554885 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:56.555373 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:56.555387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.556204 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.556212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:56.557085 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:56.557093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.557597 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.557602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:56.557769 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:56.558421 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:56.558437 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:56.560294 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:56.562027 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:56.562034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:56.562052 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:56.562423 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:56.562699 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:56.562702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:56.562704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:56.562851 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:56.563285 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:56.563293 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:56.563603 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 18:46:56.563766 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 18:46:56.564099 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":170,"threads_started":1}
I20260927 18:46:56.566620 615 heartbeater.cc:347] Connected to a master server at 127.29.159.126:43285
I20260927 18:46:56.566879 615 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:56.567374 615 heartbeater.cc:511] Master 127.29.159.126:43285 requested a full tablet report, sending...
I20260927 18:46:56.568511 32758 ts_manager.cc:194] Registered new tserver with Master: f0be148b6f4344cba5f74aedc2510ccb (127.29.159.66:36635)
I20260927 18:46:56.569092 32758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:35601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:46:56.579558 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43285
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:41463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:56.689304 620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:56.689600 620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:56.689677 620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:56.693675 620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:56.693774 620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:56.693868 620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:46:56.699482 620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:41463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:56.700671 620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:56.701875 620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:56.711560 628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:56.711568 625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:56.711719 620 server_base.cc:1034] running on GCE node
W20260927 18:46:56.711568 626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:56.712285 620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:56.712927 620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:56.714084 620 hybrid_clock.cc:648] HybridClock initialized: now 1790534816714067 us; error 37 us; skew 500 ppm
Sep 27 18:46:56 dist-test-slave-8tg2 krb5kdc[32713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534816, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:56.717919 620 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:46:56.719172 620 webserver.cc:460] Webserver started at http://127.29.159.67:41727/ using document root <none> and password file <none>
I20260927 18:46:56.719921 620 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:56.720010 620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:56.720252 620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:56.723243 620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "bc171dd1387e44c18e364146cb59489e"
format_stamp: "Formatted at 2026-09-27 18:46:56 on dist-test-slave-8tg2"
server_key: "c61ed5144b2a7cba9e8b7cb66d033d6a"
server_key_iv: "5e67c41437b13c56e72dad4dd520b00b"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.723946 620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "bc171dd1387e44c18e364146cb59489e"
format_stamp: "Formatted at 2026-09-27 18:46:56 on dist-test-slave-8tg2"
server_key: "c61ed5144b2a7cba9e8b7cb66d033d6a"
server_key_iv: "5e67c41437b13c56e72dad4dd520b00b"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.727929 620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:46:56.730435 635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:56.731746 620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:46:56.731869 620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "bc171dd1387e44c18e364146cb59489e"
format_stamp: "Formatted at 2026-09-27 18:46:56 on dist-test-slave-8tg2"
server_key: "c61ed5144b2a7cba9e8b7cb66d033d6a"
server_key_iv: "5e67c41437b13c56e72dad4dd520b00b"
server_key_version: "encryptionkey@0"
I20260927 18:46:56.731976 620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:56.748874 620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:56.751804 620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:56.751995 620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:56.752547 620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:56.753331 620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:56.753396 620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:56.753461 620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:56.753509 620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:56.762497 620 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:33059
I20260927 18:46:56.762519 748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:33059 every 8 connection(s)
I20260927 18:46:56.763468 620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:46:56.765511 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 620
I20260927 18:46:56.765662 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:46:56.765954 30333 external_mini_cluster.cc:1428] Setting key ec34ff3e61005690b4a1569c47291740
Sep 27 18:46:56 dist-test-slave-8tg2 krb5kdc[32713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534816, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:56.777048 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:56.765431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:50877 (local address 127.29.159.126:43285)
0927 18:46:56.765590 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:56.765595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:56.766621 (+ 1026us) server_negotiation.cc:407] Connection header received
0927 18:46:56.768262 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:56.768265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:56.768307 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:56.768375 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:56.769457 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.769967 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.771126 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.771330 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.771407 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:56.773836 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:56.773839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:56.773841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:56.773863 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:56.775321 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:56.775806 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:56.775808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:56.775810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:56.775882 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:56.776179 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:56.776181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:56.776182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:56.776346 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:56.776418 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:56.776841 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 18:46:56.776932 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:46:56.777822 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:56.765901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:43285 (local address 127.29.159.67:50877)
0927 18:46:56.766433 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:56.766504 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:56.768052 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:56.768514 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:56.768520 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:56.768860 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:56.769337 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:56.769350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.770082 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.770084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:56.771017 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:56.771025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:56.771497 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:56.771501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:56.771658 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:56.772275 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:56.772291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:56.773692 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:56.775413 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:56.775417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:56.775470 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:56.775722 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:56.775960 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:56.775963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:56.775964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:56.776086 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:56.776426 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:56.776430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:56.776641 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 18:46:56.776795 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 18:46:56.777027 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":158,"threads_started":1}
I20260927 18:46:56.779096 749 heartbeater.cc:347] Connected to a master server at 127.29.159.126:43285
I20260927 18:46:56.779353 749 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:56.779850 749 heartbeater.cc:511] Master 127.29.159.126:43285 requested a full tablet report, sending...
I20260927 18:46:56.780921 32758 ts_manager.cc:194] Registered new tserver with Master: bc171dd1387e44c18e364146cb59489e (127.29.159.67:33059)
I20260927 18:46:56.781445 32758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:50877
I20260927 18:46:56.791821 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:46:56.804855 330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:46:56.803253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.29.159.126:43285)
0927 18:46:56.803420 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:56.803425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:56.803636 (+ 211us) server_negotiation.cc:407] Connection header received
0927 18:46:56.803823 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:56.803827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:56.803883 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:56.803980 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:56.804714 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34280: BlockingRecv error: recv got EOF from 127.0.0.1:34280 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:46:56.806051 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 352
I20260927 18:46:56.811328 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 486
I20260927 18:46:56.816342 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 620
I20260927 18:46:56.821450 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 32726
2026-09-27T18:46:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2115 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:56 dist-test-slave-8tg2 krb5kdc[761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:56 dist-test-slave-8tg2 krb5kdc[761](info): set up 2 sockets
Sep 27 18:46:56 dist-test-slave-8tg2 krb5kdc[761](info): commencing operation
krb5kdc: starting...
W20260927 18:46:57.883391 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:46:57 dist-test-slave-8tg2 krb5kdc[761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:46:58.008824 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:34521
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:34521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:58.119753 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:58.120066 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:58.120139 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:58.124156 774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:58.124251 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:58.124286 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:58.124325 774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:58.124360 774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:58.129599 774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:34521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:34521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:58.130692 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:58.131908 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:58.140566 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:58.140637 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:58.140561 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:58.140682 774 server_base.cc:1034] running on GCE node
I20260927 18:46:58.141322 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:58.142421 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:58.143600 774 hybrid_clock.cc:648] HybridClock initialized: now 1790534818143579 us; error 36 us; skew 500 ppm
Sep 27 18:46:58 dist-test-slave-8tg2 krb5kdc[761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534818, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:58.146370 774 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:46:58.147280 774 webserver.cc:460] Webserver started at http://127.29.159.126:32999/ using document root <none> and password file <none>
I20260927 18:46:58.147928 774 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:58.148022 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:58.148239 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:58.150315 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "e5cf45a3f6794ba8a1beade58af47153"
format_stamp: "Formatted at 2026-09-27 18:46:58 on dist-test-slave-8tg2"
server_key: "c1dac25015fcf34a05f3992eb5a8f5ae"
server_key_iv: "c65013dc734241d24f914ae5daec7b5f"
server_key_version: "encryptionkey@0"
I20260927 18:46:58.150903 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "e5cf45a3f6794ba8a1beade58af47153"
format_stamp: "Formatted at 2026-09-27 18:46:58 on dist-test-slave-8tg2"
server_key: "c1dac25015fcf34a05f3992eb5a8f5ae"
server_key_iv: "c65013dc734241d24f914ae5daec7b5f"
server_key_version: "encryptionkey@0"
I20260927 18:46:58.154843 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:46:58.157428 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:58.158732 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:58.158895 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "e5cf45a3f6794ba8a1beade58af47153"
format_stamp: "Formatted at 2026-09-27 18:46:58 on dist-test-slave-8tg2"
server_key: "c1dac25015fcf34a05f3992eb5a8f5ae"
server_key_iv: "c65013dc734241d24f914ae5daec7b5f"
server_key_version: "encryptionkey@0"
I20260927 18:46:58.159018 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:58.171919 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:58.175343 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:58.175699 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:58.183208 774 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:34521
I20260927 18:46:58.183543 841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:34521 every 8 connection(s)
I20260927 18:46:58.184372 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:58.185814 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 774
I20260927 18:46:58.185945 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:46:58.186249 30333 external_mini_cluster.cc:1428] Setting key ebf0e87a3fd6d9602fd9b3049f82df84
I20260927 18:46:58.187968 842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:58.194355 842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153: Bootstrap starting.
Sep 27 18:46:58 dist-test-slave-8tg2 krb5kdc[761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:58.197659 842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:58.198511 842 log.cc:824] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:58.200966 842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153: No bootstrap required, opened a new log
I20260927 18:46:58.201344 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.187814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.29.159.126:34521)
0927 18:46:58.188310 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:58.188326 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:58.188363 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:46:58.189050 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:58.189068 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:58.189423 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:58.189797 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:58.190668 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.191759 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.192657 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.193151 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.193335 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.196362 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:58.196367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:58.196381 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:58.196412 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:58.198183 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:58.198709 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:58.198722 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:58.198726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:58.198849 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:58.199159 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:58.199162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:58.199163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:58.199730 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:58.199892 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:58.200213 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 18:46:58.200529 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":152,"threads_started":1}
I20260927 18:46:58.204314 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:46:58.204905 842 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cf45a3f6794ba8a1beade58af47153" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34521 } }
I20260927 18:46:58.205206 842 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:58.205286 842 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5cf45a3f6794ba8a1beade58af47153, State: Initialized, Role: FOLLOWER
I20260927 18:46:58.205785 842 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cf45a3f6794ba8a1beade58af47153" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34521 } }
I20260927 18:46:58.205968 842 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:58.206040 842 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:58.206147 842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:58.207170 842 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cf45a3f6794ba8a1beade58af47153" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34521 } }
I20260927 18:46:58.207577 842 leader_election.cc:302] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5cf45a3f6794ba8a1beade58af47153; no voters:
I20260927 18:46:58.207923 842 leader_election.cc:288] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:58.208006 848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:58.208221 848 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [term 1 LEADER]: Becoming Leader. State: Replica: e5cf45a3f6794ba8a1beade58af47153, State: Running, Role: LEADER
I20260927 18:46:58.208519 848 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cf45a3f6794ba8a1beade58af47153" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34521 } }
I20260927 18:46:58.209283 842 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:58.209756 850 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5cf45a3f6794ba8a1beade58af47153. Latest consensus state: current_term: 1 leader_uuid: "e5cf45a3f6794ba8a1beade58af47153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cf45a3f6794ba8a1beade58af47153" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34521 } } }
I20260927 18:46:58.209892 850 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:58.210294 852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:58.210559 849 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5cf45a3f6794ba8a1beade58af47153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5cf45a3f6794ba8a1beade58af47153" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34521 } } }
I20260927 18:46:58.210646 849 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:58.214843 852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:58.220774 852 catalog_manager.cc:1347] Generated new cluster ID: 9c7fc7c64eb24384b62717151f7dd15e
I20260927 18:46:58.220914 852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:58.230675 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:58.224834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.29.159.126:34521)
0927 18:46:58.224966 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:58.224971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:58.225253 (+ 282us) server_negotiation.cc:407] Connection header received
0927 18:46:58.225403 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:58.225406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:58.225451 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:58.225551 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:58.226135 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.226913 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.228127 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:58.228480 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:58.228570 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:58.229577 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:58.229581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:58.229584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:46:58.230171 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:58.230205 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:46:58.230239 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:58.230331 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:58.230452 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 18:46:58.230523 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:46:58.235503 852 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:58.236389 852 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:58.240160 852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5cf45a3f6794ba8a1beade58af47153: Generated new TSK 0
I20260927 18:46:58.240868 852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:46:58.253522 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 774
2026-09-27T18:46:58Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1429 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:46:58 dist-test-slave-8tg2 krb5kdc[875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:46:58 dist-test-slave-8tg2 krb5kdc[875](info): set up 2 sockets
Sep 27 18:46:58 dist-test-slave-8tg2 krb5kdc[875](info): commencing operation
krb5kdc: starting...
W20260927 18:46:59.347793 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:46:59 dist-test-slave-8tg2 krb5kdc[875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:46:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:46:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:46:59.468911 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:37215
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:37215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:59.577556 888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:59.577803 888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:59.577843 888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:59.581743 888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:46:59.581791 888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:59.581809 888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:59.581825 888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:46:59.581840 888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:46:59.586823 888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:37215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:37215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:59.587796 888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:59.588737 888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:59.597064 894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:59.597062 893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:59.597092 896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:59.597290 888 server_base.cc:1034] running on GCE node
I20260927 18:46:59.597996 888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:59.599014 888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:59.600196 888 hybrid_clock.cc:648] HybridClock initialized: now 1790534819600186 us; error 25 us; skew 500 ppm
Sep 27 18:46:59 dist-test-slave-8tg2 krb5kdc[875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534819, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:59.602962 888 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:46:59.603946 888 webserver.cc:460] Webserver started at http://127.29.159.126:40087/ using document root <none> and password file <none>
I20260927 18:46:59.604494 888 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:59.604584 888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:59.604807 888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:59.606798 888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "30d27bb5a6fe4db1a1b592989e57ea06"
format_stamp: "Formatted at 2026-09-27 18:46:59 on dist-test-slave-8tg2"
server_key: "ccddcb3686e3ad236b933d07783b0b8a"
server_key_iv: "17b58c33a8b5b38b85c44fb9b725d04b"
server_key_version: "encryptionkey@0"
I20260927 18:46:59.607362 888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "30d27bb5a6fe4db1a1b592989e57ea06"
format_stamp: "Formatted at 2026-09-27 18:46:59 on dist-test-slave-8tg2"
server_key: "ccddcb3686e3ad236b933d07783b0b8a"
server_key_iv: "17b58c33a8b5b38b85c44fb9b725d04b"
server_key_version: "encryptionkey@0"
I20260927 18:46:59.610958 888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:46:59.613336 903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:59.614523 888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:46:59.614658 888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "30d27bb5a6fe4db1a1b592989e57ea06"
format_stamp: "Formatted at 2026-09-27 18:46:59 on dist-test-slave-8tg2"
server_key: "ccddcb3686e3ad236b933d07783b0b8a"
server_key_iv: "17b58c33a8b5b38b85c44fb9b725d04b"
server_key_version: "encryptionkey@0"
I20260927 18:46:59.614768 888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:59.651379 888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:59.654508 888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:59.654713 888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:59.661954 888 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:37215
I20260927 18:46:59.662240 955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:37215 every 8 connection(s)
I20260927 18:46:59.662935 888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:46:59.665343 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 888
I20260927 18:46:59.665460 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:46:59.665761 30333 external_mini_cluster.cc:1428] Setting key e6f7e11cacc9870941b9172d521121a0
I20260927 18:46:59.665823 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:46:59.672864 956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06: Bootstrap starting.
Sep 27 18:46:59 dist-test-slave-8tg2 krb5kdc[875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:46:59.675269 956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06: Neither blocks nor log segments found. Creating new log.
I20260927 18:46:59.676111 956 log.cc:824] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06: Log is configured to *not* fsync() on all Append() calls
I20260927 18:46:59.677978 956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06: No bootstrap required, opened a new log
I20260927 18:46:59.680307 959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:59.667152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.29.159.126:37215)
0927 18:46:59.667573 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:59.667595 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:59.667639 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:46:59.668524 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:59.668547 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:59.668961 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:59.669341 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:59.669954 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.671460 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.672327 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.672993 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.673142 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:59.675843 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:59.675848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:59.675861 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:59.675888 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:59.677574 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:59.678057 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:59.678064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:59.678073 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:59.678202 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:59.678494 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:59.678496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:59.678497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:59.678873 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:59.679017 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:59.679280 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 18:46:59.679596 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
I20260927 18:46:59.681918 956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d27bb5a6fe4db1a1b592989e57ea06" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37215 } }
I20260927 18:46:59.682168 956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:46:59.682226 956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d27bb5a6fe4db1a1b592989e57ea06, State: Initialized, Role: FOLLOWER
I20260927 18:46:59.682650 956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d27bb5a6fe4db1a1b592989e57ea06" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37215 } }
I20260927 18:46:59.682765 956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:46:59.682811 956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:46:59.682878 956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:46:59.683689 956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d27bb5a6fe4db1a1b592989e57ea06" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37215 } }
I20260927 18:46:59.683971 956 leader_election.cc:302] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d27bb5a6fe4db1a1b592989e57ea06; no voters:
I20260927 18:46:59.684201 956 leader_election.cc:288] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:46:59.684342 961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:46:59.684595 961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [term 1 LEADER]: Becoming Leader. State: Replica: 30d27bb5a6fe4db1a1b592989e57ea06, State: Running, Role: LEADER
I20260927 18:46:59.684898 961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d27bb5a6fe4db1a1b592989e57ea06" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37215 } }
I20260927 18:46:59.685340 956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:46:59.686617 962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30d27bb5a6fe4db1a1b592989e57ea06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d27bb5a6fe4db1a1b592989e57ea06" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37215 } } }
I20260927 18:46:59.686744 962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:59.686599 963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d27bb5a6fe4db1a1b592989e57ea06. Latest consensus state: current_term: 1 leader_uuid: "30d27bb5a6fe4db1a1b592989e57ea06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d27bb5a6fe4db1a1b592989e57ea06" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37215 } } }
I20260927 18:46:59.687003 963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06 [sys.catalog]: This master's current role is: LEADER
I20260927 18:46:59.687721 970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:46:59.690726 970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:46:59.696636 970 catalog_manager.cc:1347] Generated new cluster ID: ce0812e5695a4b378050bc8ebee17583
I20260927 18:46:59.696709 970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:46:59.703636 970 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:46:59.704499 970 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:46:59.710120 970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30d27bb5a6fe4db1a1b592989e57ea06: Generated new TSK 0
I20260927 18:46:59.710950 970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:46:59.738323 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37215
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:46:59.849890 982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:46:59.850190 982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:46:59.850271 982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:46:59.854254 982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:46:59.854343 982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:46:59.854435 982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:46:59.859715 982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:46:59.860790 982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:46:59.861879 982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:46:59.871560 988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:59.871636 982 server_base.cc:1034] running on GCE node
W20260927 18:46:59.871532 990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:46:59.871826 987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:46:59.872236 982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:46:59.872897 982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:46:59.874060 982 hybrid_clock.cc:648] HybridClock initialized: now 1790534819874037 us; error 38 us; skew 500 ppm
Sep 27 18:46:59 dist-test-slave-8tg2 krb5kdc[875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534819, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:46:59.877007 982 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:46:59.877857 982 webserver.cc:460] Webserver started at http://127.29.159.65:33981/ using document root <none> and password file <none>
I20260927 18:46:59.878440 982 fs_manager.cc:359] Metadata directory not provided
I20260927 18:46:59.878501 982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:46:59.878722 982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:46:59.880810 982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "580ab94a48d24c0a9b508bb3bcbc078a"
format_stamp: "Formatted at 2026-09-27 18:46:59 on dist-test-slave-8tg2"
server_key: "972e46183673c966d76ec39fbf1d95b0"
server_key_iv: "dbc630482050cc3c2d4d9f1c0bc6fd09"
server_key_version: "encryptionkey@0"
I20260927 18:46:59.881371 982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "580ab94a48d24c0a9b508bb3bcbc078a"
format_stamp: "Formatted at 2026-09-27 18:46:59 on dist-test-slave-8tg2"
server_key: "972e46183673c966d76ec39fbf1d95b0"
server_key_iv: "dbc630482050cc3c2d4d9f1c0bc6fd09"
server_key_version: "encryptionkey@0"
I20260927 18:46:59.885056 982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 18:46:59.887667 997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:59.888873 982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:46:59.889004 982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "580ab94a48d24c0a9b508bb3bcbc078a"
format_stamp: "Formatted at 2026-09-27 18:46:59 on dist-test-slave-8tg2"
server_key: "972e46183673c966d76ec39fbf1d95b0"
server_key_iv: "dbc630482050cc3c2d4d9f1c0bc6fd09"
server_key_version: "encryptionkey@0"
I20260927 18:46:59.889114 982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:46:59.905133 982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:46:59.908551 982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:46:59.908771 982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:46:59.909384 982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:46:59.910218 982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:46:59.910300 982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:59.910377 982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:46:59.910429 982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:46:59.919556 982 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:36393
I20260927 18:46:59.919590 1110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:36393 every 8 connection(s)
I20260927 18:46:59.920442 982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:46:59.924919 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 982
I20260927 18:46:59.925078 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:46:59.925462 30333 external_mini_cluster.cc:1428] Setting key bd046c321c59e34cfd44e9b59537bf9a
Sep 27 18:46:59 dist-test-slave-8tg2 krb5kdc[875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534819, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:46:59.936666 959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:59.922492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:34795 (local address 127.29.159.126:37215)
0927 18:46:59.922636 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 18:46:59.922640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:46:59.923627 (+ 987us) server_negotiation.cc:407] Connection header received
0927 18:46:59.925101 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:46:59.925105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:46:59.925180 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:46:59.925266 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:46:59.926728 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.927366 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.928585 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.928832 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.928948 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:59.932279 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:46:59.932282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:46:59.932284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:46:59.932307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:46:59.933966 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:59.934676 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:59.934679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:59.934680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:59.934761 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:46:59.935254 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:46:59.935260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:46:59.935261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:46:59.935525 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:46:59.935611 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:46:59.936340 (+ 729us) server_negotiation.cc:299] Negotiation successful
0927 18:46:59.936461 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:46:59.937733 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:46:59.923004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:37215 (local address 127.29.159.65:34795)
0927 18:46:59.923486 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 18:46:59.923529 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:46:59.924873 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:46:59.925528 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:46:59.925557 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:46:59.926029 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:46:59.926569 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:59.926583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.927566 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.927569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:59.928453 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:46:59.928462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:46:59.928985 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:46:59.928990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:46:59.929252 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:46:59.929911 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:46:59.929928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:46:59.932068 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:46:59.934102 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:59.934110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:59.934139 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:59.934556 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:59.934888 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:46:59.934892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:46:59.934894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:46:59.935106 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:46:59.935646 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:46:59.935656 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:46:59.936022 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 18:46:59.936279 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 18:46:59.936726 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
I20260927 18:46:59.939353 1111 heartbeater.cc:347] Connected to a master server at 127.29.159.126:37215
I20260927 18:46:59.939646 1111 heartbeater.cc:464] Registering TS with master...
I20260927 18:46:59.940188 1111 heartbeater.cc:511] Master 127.29.159.126:37215 requested a full tablet report, sending...
I20260927 18:46:59.941902 920 ts_manager.cc:194] Registered new tserver with Master: 580ab94a48d24c0a9b508bb3bcbc078a (127.29.159.65:36393)
I20260927 18:46:59.943152 920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:34795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:46:59.955868 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37215
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:00.069118 1116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:00.069484 1116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:00.069569 1116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:00.074090 1116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:00.074185 1116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:00.074285 1116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:00.079821 1116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:00.080966 1116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:00.082028 1116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:00.091985 1121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:00.092038 1116 server_base.cc:1034] running on GCE node
W20260927 18:47:00.092051 1122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:00.092000 1124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:00.092612 1116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:00.093319 1116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:00.094551 1116 hybrid_clock.cc:648] HybridClock initialized: now 1790534820094524 us; error 44 us; skew 500 ppm
Sep 27 18:47:00 dist-test-slave-8tg2 krb5kdc[875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534820, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:00.097656 1116 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:47:00.098646 1116 webserver.cc:460] Webserver started at http://127.29.159.66:46703/ using document root <none> and password file <none>
I20260927 18:47:00.099234 1116 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:00.099335 1116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:00.099606 1116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:00.102149 1116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "a35de12c04594ca29c0ec03c0411cd89"
format_stamp: "Formatted at 2026-09-27 18:47:00 on dist-test-slave-8tg2"
server_key: "79c082b0a5db323646b72fa435240606"
server_key_iv: "7d151ec1d6a6bbfa56c00f3342205236"
server_key_version: "encryptionkey@0"
I20260927 18:47:00.102761 1116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "a35de12c04594ca29c0ec03c0411cd89"
format_stamp: "Formatted at 2026-09-27 18:47:00 on dist-test-slave-8tg2"
server_key: "79c082b0a5db323646b72fa435240606"
server_key_iv: "7d151ec1d6a6bbfa56c00f3342205236"
server_key_version: "encryptionkey@0"
I20260927 18:47:00.106873 1116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 18:47:00.109513 1131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:00.110832 1116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:00.110982 1116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "a35de12c04594ca29c0ec03c0411cd89"
format_stamp: "Formatted at 2026-09-27 18:47:00 on dist-test-slave-8tg2"
server_key: "79c082b0a5db323646b72fa435240606"
server_key_iv: "7d151ec1d6a6bbfa56c00f3342205236"
server_key_version: "encryptionkey@0"
I20260927 18:47:00.111109 1116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:00.135308 1116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:00.138883 1116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:00.139098 1116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:00.139755 1116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:00.140617 1116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:00.140692 1116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:00.140765 1116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:00.140808 1116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:00.149874 1116 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:39927
I20260927 18:47:00.149899 1244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:39927 every 8 connection(s)
I20260927 18:47:00.150810 1116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:00.152642 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 1116
I20260927 18:47:00.152772 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:00.153061 30333 external_mini_cluster.cc:1428] Setting key 53eaa89a8ff1181c6c9d058e1f0e2c2c
Sep 27 18:47:00 dist-test-slave-8tg2 krb5kdc[875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534820, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:47:00.165890 959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:00.152712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:37773 (local address 127.29.159.126:37215)
0927 18:47:00.152885 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:00.152889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:00.153765 (+ 876us) server_negotiation.cc:407] Connection header received
0927 18:47:00.155141 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:00.155148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:00.155211 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:00.155304 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:00.156813 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:00.157335 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:00.159145 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:00.159378 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:00.159544 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:00.162347 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:00.162350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:00.162352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:00.162373 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:00.163938 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:00.164411 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:00.164414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:00.164415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:00.164487 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:00.164841 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:00.164843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:00.164844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:00.165011 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:00.165088 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:00.165636 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 18:47:00.165771 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:47:00.167055 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:00.153123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:37215 (local address 127.29.159.66:37773)
0927 18:47:00.153605 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:00.153658 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:00.154930 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:00.155508 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:00.155515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:00.155860 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:00.156653 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:00.156675 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:00.157459 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:00.157465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:00.159022 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:00.159035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:00.159585 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:00.159591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:00.159838 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:00.160475 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:00.160501 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:00.162166 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:00.164036 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:00.164040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:00.164052 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:00.164317 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:00.164566 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:00.164568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:00.164569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:00.164738 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:00.165099 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:00.165104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:00.165378 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 18:47:00.165575 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 18:47:00.165917 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":120,"threads_started":1}
I20260927 18:47:00.168411 1245 heartbeater.cc:347] Connected to a master server at 127.29.159.126:37215
I20260927 18:47:00.168661 1245 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:00.169252 1245 heartbeater.cc:511] Master 127.29.159.126:37215 requested a full tablet report, sending...
I20260927 18:47:00.170441 920 ts_manager.cc:194] Registered new tserver with Master: a35de12c04594ca29c0ec03c0411cd89 (127.29.159.66:39927)
I20260927 18:47:00.171345 920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:37773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:47:00.182610 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37215
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:00.295274 1250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:00.295612 1250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:00.295661 1250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:00.299749 1250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:00.299810 1250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:00.299875 1250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:00.305327 1250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:00.306350 1250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:00.307350 1250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:00.317318 1255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:00.317449 1250 server_base.cc:1034] running on GCE node
W20260927 18:47:00.317320 1258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:00.317318 1256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:00.318017 1250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:00.318663 1250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:00.319825 1250 hybrid_clock.cc:648] HybridClock initialized: now 1790534820319802 us; error 35 us; skew 500 ppm
Sep 27 18:47:00 dist-test-slave-8tg2 krb5kdc[875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534820, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:00.322556 1250 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:47:00.323540 1250 webserver.cc:460] Webserver started at http://127.29.159.67:33123/ using document root <none> and password file <none>
I20260927 18:47:00.324105 1250 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:00.324204 1250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:00.324433 1250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:00.326475 1250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "1515d7f4974c472383d2387fcda4d05a"
format_stamp: "Formatted at 2026-09-27 18:47:00 on dist-test-slave-8tg2"
server_key: "17d1b807f3ff28f14b2ab839be92c209"
server_key_iv: "7d53e8087f8431c14b69f23e524adf64"
server_key_version: "encryptionkey@0"
I20260927 18:47:00.327018 1250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "1515d7f4974c472383d2387fcda4d05a"
format_stamp: "Formatted at 2026-09-27 18:47:00 on dist-test-slave-8tg2"
server_key: "17d1b807f3ff28f14b2ab839be92c209"
server_key_iv: "7d53e8087f8431c14b69f23e524adf64"
server_key_version: "encryptionkey@0"
I20260927 18:47:00.330828 1250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:47:00.333333 1265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:00.334581 1250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:00.334731 1250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "1515d7f4974c472383d2387fcda4d05a"
format_stamp: "Formatted at 2026-09-27 18:47:00 on dist-test-slave-8tg2"
server_key: "17d1b807f3ff28f14b2ab839be92c209"
server_key_iv: "7d53e8087f8431c14b69f23e524adf64"
server_key_version: "encryptionkey@0"
I20260927 18:47:00.334859 1250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:00.352741 1250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:00.355672 1250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:00.355875 1250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:00.356468 1250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:00.357291 1250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:00.357360 1250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:00.357431 1250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:00.357482 1250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:00.366601 1250 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:35507
I20260927 18:47:00.366627 1378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:35507 every 8 connection(s)
I20260927 18:47:00.367538 1250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:00.368567 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 1250
I20260927 18:47:00.368716 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:00.369108 30333 external_mini_cluster.cc:1428] Setting key 3dfb922dd9d502db6100921394b8e823
Sep 27 18:47:00 dist-test-slave-8tg2 krb5kdc[875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534820, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:00.381325 959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:00.369566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:43743 (local address 127.29.159.126:37215)
0927 18:47:00.370933 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:00.370936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:00.370949 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:47:00.372395 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:00.372397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:00.372442 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:00.372511 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:00.373794 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:00.374307 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:00.375400 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:00.375716 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:00.375805 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:00.378268 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:00.378271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:00.378273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:00.378296 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:00.379570 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:00.380018 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:00.380020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:00.380021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:00.380089 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:00.380382 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:00.380384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:00.380385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:00.380547 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:00.380625 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:00.381102 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 18:47:00.381198 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1291}
I20260927 18:47:00.382239 1381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:00.370006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:37215 (local address 127.29.159.67:43743)
0927 18:47:00.370574 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:00.370630 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:00.372195 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:00.372808 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:00.372815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:00.373165 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:00.373670 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:00.373684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:00.374413 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:00.374416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:00.375298 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:00.375307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:00.375875 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:00.375879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:00.376023 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:00.376650 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:00.376666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:00.378114 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:00.379665 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:00.379669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:00.379681 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:00.379936 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:00.380164 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:00.380166 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:00.380167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:00.380298 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:00.380668 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:00.380674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:00.380895 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 18:47:00.381057 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 18:47:00.381313 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":134,"threads_started":1}
I20260927 18:47:00.383311 1379 heartbeater.cc:347] Connected to a master server at 127.29.159.126:37215
I20260927 18:47:00.383574 1379 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:00.384037 1379 heartbeater.cc:511] Master 127.29.159.126:37215 requested a full tablet report, sending...
I20260927 18:47:00.385008 920 ts_manager.cc:194] Registered new tserver with Master: 1515d7f4974c472383d2387fcda4d05a (127.29.159.67:35507)
I20260927 18:47:00.385576 920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:43743
I20260927 18:47:00.395236 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:47:00 dist-test-slave-8tg2 krb5kdc[875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534820, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 18:47:00 dist-test-slave-8tg2 krb5kdc[875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534820, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:00.425632 959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:00.412333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.29.159.126:37215)
0927 18:47:00.412517 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:00.412521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:00.412606 (+ 85us) server_negotiation.cc:407] Connection header received
0927 18:47:00.412715 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:00.412717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:00.412761 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:00.412834 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:00.413499 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:00.414097 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:00.415057 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:00.415276 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:00.415368 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:00.422389 (+ 7021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:00.422392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:00.422394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:00.422422 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:00.423947 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:00.424356 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:00.424359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:00.424360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:00.424432 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:00.424675 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:00.424677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:00.424678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:00.424889 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:00.424991 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:00.425314 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 18:47:00.425410 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 18:47:00.426023 920 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59642
I20260927 18:47:00.426903 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 982
I20260927 18:47:00.432462 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 1116
I20260927 18:47:00.437767 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 1250
I20260927 18:47:00.443163 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 888
2026-09-27T18:47:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2191 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:47:00 dist-test-slave-8tg2 krb5kdc[1391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:47:00 dist-test-slave-8tg2 krb5kdc[1391](info): set up 2 sockets
Sep 27 18:47:00 dist-test-slave-8tg2 krb5kdc[1391](info): commencing operation
krb5kdc: starting...
W20260927 18:47:01.500155 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:47:01 dist-test-slave-8tg2 krb5kdc[1391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:47:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:47:01.626165 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:34625
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:34625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:01.735985 1404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:01.736253 1404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:01.736304 1404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:01.740260 1404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:01.740314 1404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:01.740332 1404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:01.740350 1404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:01.740365 1404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:01.746127 1404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:34625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:34625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:01.747119 1404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:01.748188 1404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:01.756799 1410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:01.756794 1409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:01.756848 1404 server_base.cc:1034] running on GCE node
W20260927 18:47:01.756794 1412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:01.757589 1404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:01.758756 1404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:01.759948 1404 hybrid_clock.cc:648] HybridClock initialized: now 1790534821759931 us; error 28 us; skew 500 ppm
Sep 27 18:47:01 dist-test-slave-8tg2 krb5kdc[1391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534821, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:01.767400 1404 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:47:01.768474 1404 webserver.cc:460] Webserver started at http://127.29.159.126:33753/ using document root <none> and password file <none>
I20260927 18:47:01.769078 1404 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:01.769171 1404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:01.769409 1404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:01.771562 1404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "90ae3f89167845de929a11a0309f0d36"
format_stamp: "Formatted at 2026-09-27 18:47:01 on dist-test-slave-8tg2"
server_key: "9fce9e47a3bb1272b0b26a7e9f99ebc5"
server_key_iv: "de140a2d77cac260087a64ed84d70297"
server_key_version: "encryptionkey@0"
I20260927 18:47:01.772131 1404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "90ae3f89167845de929a11a0309f0d36"
format_stamp: "Formatted at 2026-09-27 18:47:01 on dist-test-slave-8tg2"
server_key: "9fce9e47a3bb1272b0b26a7e9f99ebc5"
server_key_iv: "de140a2d77cac260087a64ed84d70297"
server_key_version: "encryptionkey@0"
I20260927 18:47:01.775933 1404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:47:01.778407 1419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:01.779659 1404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:01.779827 1404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "90ae3f89167845de929a11a0309f0d36"
format_stamp: "Formatted at 2026-09-27 18:47:01 on dist-test-slave-8tg2"
server_key: "9fce9e47a3bb1272b0b26a7e9f99ebc5"
server_key_iv: "de140a2d77cac260087a64ed84d70297"
server_key_version: "encryptionkey@0"
I20260927 18:47:01.779946 1404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:01.807669 1404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:01.811196 1404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:01.811439 1404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:01.818598 1404 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:34625
I20260927 18:47:01.818611 1471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:34625 every 8 connection(s)
I20260927 18:47:01.819583 1404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:01.822443 1472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:01.822791 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 1404
I20260927 18:47:01.822932 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:01.823210 30333 external_mini_cluster.cc:1428] Setting key b5e4b46d899138589a984054b5b3c1ef
I20260927 18:47:01.829527 1472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36: Bootstrap starting.
Sep 27 18:47:01 dist-test-slave-8tg2 krb5kdc[1391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:01.832365 1472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:01.833087 1472 log.cc:824] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:01.835237 1472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36: No bootstrap required, opened a new log
I20260927 18:47:01.837819 1475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:01.824666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.29.159.126:34625)
0927 18:47:01.825092 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:01.825110 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:01.825146 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:47:01.825806 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:01.825823 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:01.826159 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:01.826454 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:01.827409 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:01.828414 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:01.829549 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:01.830070 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:01.830186 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:01.832896 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:01.832901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:01.832914 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:01.832941 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:01.834865 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:01.835338 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:01.835366 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:01.835375 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:01.835531 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:01.835826 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:01.835831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:01.835832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:01.836230 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:01.836427 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:01.836747 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 18:47:01.837088 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":134,"threads_started":1}
I20260927 18:47:01.839210 1472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ae3f89167845de929a11a0309f0d36" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34625 } }
I20260927 18:47:01.839501 1472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:01.839557 1472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ae3f89167845de929a11a0309f0d36, State: Initialized, Role: FOLLOWER
I20260927 18:47:01.840037 1472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ae3f89167845de929a11a0309f0d36" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34625 } }
I20260927 18:47:01.840221 1472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:01.840277 1472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:01.840350 1472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:01.841276 1472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ae3f89167845de929a11a0309f0d36" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34625 } }
I20260927 18:47:01.841679 1472 leader_election.cc:302] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90ae3f89167845de929a11a0309f0d36; no voters:
I20260927 18:47:01.842052 1472 leader_election.cc:288] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:01.842177 1477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:01.842420 1477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [term 1 LEADER]: Becoming Leader. State: Replica: 90ae3f89167845de929a11a0309f0d36, State: Running, Role: LEADER
I20260927 18:47:01.842777 1477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ae3f89167845de929a11a0309f0d36" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34625 } }
I20260927 18:47:01.843427 1472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:01.844548 1478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90ae3f89167845de929a11a0309f0d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ae3f89167845de929a11a0309f0d36" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34625 } } }
I20260927 18:47:01.844681 1478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:01.845010 1479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90ae3f89167845de929a11a0309f0d36. Latest consensus state: current_term: 1 leader_uuid: "90ae3f89167845de929a11a0309f0d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ae3f89167845de929a11a0309f0d36" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34625 } } }
I20260927 18:47:01.845105 1486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:01.845168 1479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:01.848655 1486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:01.854625 1486 catalog_manager.cc:1347] Generated new cluster ID: 6006af6cc45b45b19c51b1a7d2cff785
I20260927 18:47:01.854724 1486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:01.869608 1486 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:01.870656 1486 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:01.878420 1486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90ae3f89167845de929a11a0309f0d36: Generated new TSK 0
I20260927 18:47:01.879123 1486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:47:01.911178 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34625
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:02.048015 1498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:02.048321 1498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:02.048393 1498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:02.052542 1498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:02.052628 1498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:02.052716 1498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:02.058121 1498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:02.059207 1498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:02.060307 1498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:02.070128 1506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:02.070164 1503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:02.070252 1498 server_base.cc:1034] running on GCE node
W20260927 18:47:02.070160 1504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:02.070788 1498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:02.071488 1498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:02.072666 1498 hybrid_clock.cc:648] HybridClock initialized: now 1790534822072631 us; error 51 us; skew 500 ppm
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534822, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:02.075649 1498 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:47:02.076507 1498 webserver.cc:460] Webserver started at http://127.29.159.65:37927/ using document root <none> and password file <none>
I20260927 18:47:02.077040 1498 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:02.077098 1498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:02.077261 1498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:02.079361 1498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "aebda87ada6f4d86926d97f8d6ed2b13"
format_stamp: "Formatted at 2026-09-27 18:47:02 on dist-test-slave-8tg2"
server_key: "0c776a1e89bc06dc0196ef43bd193fe7"
server_key_iv: "6fba953a27515b026200eb6153c052fd"
server_key_version: "encryptionkey@0"
I20260927 18:47:02.079962 1498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "aebda87ada6f4d86926d97f8d6ed2b13"
format_stamp: "Formatted at 2026-09-27 18:47:02 on dist-test-slave-8tg2"
server_key: "0c776a1e89bc06dc0196ef43bd193fe7"
server_key_iv: "6fba953a27515b026200eb6153c052fd"
server_key_version: "encryptionkey@0"
I20260927 18:47:02.083860 1498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:47:02.086490 1513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:02.087894 1498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:47:02.088013 1498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "aebda87ada6f4d86926d97f8d6ed2b13"
format_stamp: "Formatted at 2026-09-27 18:47:02 on dist-test-slave-8tg2"
server_key: "0c776a1e89bc06dc0196ef43bd193fe7"
server_key_iv: "6fba953a27515b026200eb6153c052fd"
server_key_version: "encryptionkey@0"
I20260927 18:47:02.088132 1498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:02.106874 1498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:02.110641 1498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:02.110885 1498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:02.111562 1498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:02.112399 1498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:02.112447 1498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:02.112509 1498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:02.112543 1498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:02.121942 1498 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:33921
I20260927 18:47:02.121966 1626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:33921 every 8 connection(s)
I20260927 18:47:02.122864 1498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:02.128582 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 1498
I20260927 18:47:02.128708 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:47:02.128999 30333 external_mini_cluster.cc:1428] Setting key 265d4034a3962cf62bbcc569973315cd
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534822, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:02.138283 1475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:02.124737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:37063 (local address 127.29.159.126:34625)
0927 18:47:02.124879 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:02.124882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:02.125741 (+ 859us) server_negotiation.cc:407] Connection header received
0927 18:47:02.127149 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:02.127152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:02.127206 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:02.127296 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:02.128559 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.129286 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.130762 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.130985 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.131066 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:02.133761 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:02.133767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:02.133774 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:02.133807 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:02.135990 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:02.136549 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:02.136555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:02.136560 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:02.136671 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:02.137029 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:02.137036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:02.137041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:02.137306 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:02.137423 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:02.137907 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 18:47:02.138048 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:47:02.139042 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:02.125103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:34625 (local address 127.29.159.65:37063)
0927 18:47:02.125586 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:02.125629 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:02.126936 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:02.127482 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:02.127489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:02.127857 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:02.128381 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:02.128402 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.129447 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.129450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:02.130641 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:02.130652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.131148 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.131155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:02.131397 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:02.132077 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:02.132104 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:02.133576 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:02.136126 (+ 2550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:02.136131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:02.136145 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:02.136436 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:02.136775 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:02.136778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:02.136780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:02.136923 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:02.137410 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:02.137415 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:02.137660 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 18:47:02.137822 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 18:47:02.138084 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":114,"threads_started":1}
I20260927 18:47:02.140398 1627 heartbeater.cc:347] Connected to a master server at 127.29.159.126:34625
I20260927 18:47:02.140673 1627 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:02.141275 1627 heartbeater.cc:511] Master 127.29.159.126:34625 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
I20260927 18:47:02.142854 1436 ts_manager.cc:194] Registered new tserver with Master: aebda87ada6f4d86926d97f8d6ed2b13 (127.29.159.65:33921)
I20260927 18:47:02.144213 1436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:37063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:47:02.157837 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34625
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:02.269735 1632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:02.270042 1632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:02.270088 1632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:02.274274 1632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:02.274358 1632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:02.274421 1632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:02.280224 1632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:02.281320 1632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:02.282349 1632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:02.292358 1640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:02.292409 1638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:02.292363 1637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:02.292692 1632 server_base.cc:1034] running on GCE node
I20260927 18:47:02.293262 1632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:02.293984 1632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:02.295182 1632 hybrid_clock.cc:648] HybridClock initialized: now 1790534822295139 us; error 50 us; skew 500 ppm
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534822, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:02.298002 1632 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:47:02.298857 1632 webserver.cc:460] Webserver started at http://127.29.159.66:40885/ using document root <none> and password file <none>
I20260927 18:47:02.299398 1632 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:02.299518 1632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:02.299706 1632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:02.301712 1632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "f560ce0f18164bf1b7324764183023ed"
format_stamp: "Formatted at 2026-09-27 18:47:02 on dist-test-slave-8tg2"
server_key: "95aff54f82882243c06239499760e8ba"
server_key_iv: "a77e059652f3c0dcbfb64983252847ba"
server_key_version: "encryptionkey@0"
I20260927 18:47:02.302237 1632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "f560ce0f18164bf1b7324764183023ed"
format_stamp: "Formatted at 2026-09-27 18:47:02 on dist-test-slave-8tg2"
server_key: "95aff54f82882243c06239499760e8ba"
server_key_iv: "a77e059652f3c0dcbfb64983252847ba"
server_key_version: "encryptionkey@0"
I20260927 18:47:02.305956 1632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 18:47:02.308434 1647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:02.309721 1632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:02.309859 1632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "f560ce0f18164bf1b7324764183023ed"
format_stamp: "Formatted at 2026-09-27 18:47:02 on dist-test-slave-8tg2"
server_key: "95aff54f82882243c06239499760e8ba"
server_key_iv: "a77e059652f3c0dcbfb64983252847ba"
server_key_version: "encryptionkey@0"
I20260927 18:47:02.309966 1632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:02.324733 1632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:02.328323 1632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:02.328481 1632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:02.329137 1632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:02.329988 1632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:02.330034 1632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:02.330102 1632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:02.330139 1632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:02.339213 1632 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:33157
I20260927 18:47:02.339272 1760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:33157 every 8 connection(s)
I20260927 18:47:02.340222 1632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:02.343854 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 1632
I20260927 18:47:02.343991 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:02.344302 30333 external_mini_cluster.cc:1428] Setting key bf85df65a8a20869ea481363bd4ac290
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534822, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:47:02.357144 1475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:02.342200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:34555 (local address 127.29.159.126:34625)
0927 18:47:02.342383 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:02.342388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:02.343198 (+ 810us) server_negotiation.cc:407] Connection header received
0927 18:47:02.344939 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:02.344943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:02.345023 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:02.345122 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:02.346448 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.347306 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.349038 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.349536 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.349653 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:02.352298 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:02.352302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:02.352305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:02.352335 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:02.354277 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:02.355190 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:02.355195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:02.355197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:02.355389 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:02.355928 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:02.355934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:02.355935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:02.356185 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:02.356270 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:02.356816 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 18:47:02.357005 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:47:02.358090 1763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:02.342601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:34625 (local address 127.29.159.66:34555)
0927 18:47:02.343037 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:02.343078 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:02.344692 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:02.345321 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:02.345329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:02.345819 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:02.346304 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:02.346317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.347466 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.347471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:02.348890 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:02.348900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.349691 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.349693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:02.349841 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:02.350522 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:02.350538 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:02.352066 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:02.354422 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:02.354430 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:02.354488 (+ 58us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:02.355037 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:02.355585 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:02.355590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:02.355592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:02.355816 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:02.356285 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:02.356290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:02.356524 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 18:47:02.356699 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 18:47:02.357080 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260927 18:47:02.359354 1761 heartbeater.cc:347] Connected to a master server at 127.29.159.126:34625
I20260927 18:47:02.359663 1761 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:02.360283 1761 heartbeater.cc:511] Master 127.29.159.126:34625 requested a full tablet report, sending...
I20260927 18:47:02.361337 1436 ts_manager.cc:194] Registered new tserver with Master: f560ce0f18164bf1b7324764183023ed (127.29.159.66:33157)
I20260927 18:47:02.362202 1436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:34555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:47:02.374300 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34625
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:02.483911 1766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:02.484160 1766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:02.484247 1766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:02.488279 1766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:02.488361 1766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:02.488448 1766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:02.493710 1766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:02.494717 1766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:02.495783 1766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:02.504971 1771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:02.504971 1772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:02.505035 1774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:02.505218 1766 server_base.cc:1034] running on GCE node
I20260927 18:47:02.505577 1766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:02.506256 1766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:02.507467 1766 hybrid_clock.cc:648] HybridClock initialized: now 1790534822507404 us; error 28 us; skew 500 ppm
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534822, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:02.510354 1766 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:47:02.511268 1766 webserver.cc:460] Webserver started at http://127.29.159.67:35745/ using document root <none> and password file <none>
I20260927 18:47:02.511884 1766 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:02.511946 1766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:02.512197 1766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:02.514248 1766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "b398048ac1fc452885881ee8dccc039b"
format_stamp: "Formatted at 2026-09-27 18:47:02 on dist-test-slave-8tg2"
server_key: "e45f472661be92f92d1d47213648d07c"
server_key_iv: "9826d08f4daef34ea03f99c1d97bd416"
server_key_version: "encryptionkey@0"
I20260927 18:47:02.514796 1766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "b398048ac1fc452885881ee8dccc039b"
format_stamp: "Formatted at 2026-09-27 18:47:02 on dist-test-slave-8tg2"
server_key: "e45f472661be92f92d1d47213648d07c"
server_key_iv: "9826d08f4daef34ea03f99c1d97bd416"
server_key_version: "encryptionkey@0"
I20260927 18:47:02.518378 1766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 18:47:02.520819 1781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:02.522053 1766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:47:02.522213 1766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "b398048ac1fc452885881ee8dccc039b"
format_stamp: "Formatted at 2026-09-27 18:47:02 on dist-test-slave-8tg2"
server_key: "e45f472661be92f92d1d47213648d07c"
server_key_iv: "9826d08f4daef34ea03f99c1d97bd416"
server_key_version: "encryptionkey@0"
I20260927 18:47:02.522349 1766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:02.544876 1766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:02.548028 1766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:02.548244 1766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:02.548853 1766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:02.549677 1766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:02.549747 1766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:02.549820 1766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:02.549846 1766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:02.559362 1766 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:32967
I20260927 18:47:02.559387 1894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:32967 every 8 connection(s)
I20260927 18:47:02.560355 1766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534822, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:02.570406 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 1766
I20260927 18:47:02.570582 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:02.570885 30333 external_mini_cluster.cc:1428] Setting key ce756d0c4b94b8d307376d0b1c62fa56
I20260927 18:47:02.574611 1475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:02.562186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:50055 (local address 127.29.159.126:34625)
0927 18:47:02.562334 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:02.562338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:02.563128 (+ 790us) server_negotiation.cc:407] Connection header received
0927 18:47:02.564595 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:02.564598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:02.564656 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:02.564752 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:02.565914 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.566436 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.567592 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.567811 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.567918 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:02.570534 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:02.570541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:02.570544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:02.570578 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:02.572317 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:02.572974 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:02.572977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:02.572978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:02.573086 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:02.573506 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:02.573509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:02.573510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:02.573703 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:02.573792 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:02.574391 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 18:47:02.574494 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:47:02.575580 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:02.562554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:34625 (local address 127.29.159.67:50055)
0927 18:47:02.562982 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:02.563022 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:02.564368 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:02.564929 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:02.564937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:02.565275 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:02.565765 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:02.565779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.566557 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.566563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:02.567481 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:02.567490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.567977 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.567981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:02.568152 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:02.568729 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:02.568745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:02.570307 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:02.572459 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:02.572466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:02.572507 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:02.572868 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:02.573211 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:02.573215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:02.573218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:02.573408 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:02.573906 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:02.573912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:02.574154 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 18:47:02.574328 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 18:47:02.574591 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260927 18:47:02.576828 1895 heartbeater.cc:347] Connected to a master server at 127.29.159.126:34625
I20260927 18:47:02.577075 1895 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:02.577566 1895 heartbeater.cc:511] Master 127.29.159.126:34625 requested a full tablet report, sending...
I20260927 18:47:02.578576 1436 ts_manager.cc:194] Registered new tserver with Master: b398048ac1fc452885881ee8dccc039b (127.29.159.67:32967)
I20260927 18:47:02.579106 1436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:50055
I20260927 18:47:02.584940 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:47:02.610351 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:02.600995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.29.159.65:33921)
0927 18:47:02.601332 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:02.601347 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:02.601371 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:47:02.601422 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:02.601427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:02.601617 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:02.601738 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:02.602385 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.603143 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.604057 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.604439 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.604534 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:02.606858 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:02.606866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:02.606879 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:02.606901 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:02.608555 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:02.608942 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:02.608947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:02.608948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:02.609024 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:02.609393 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:02.609399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:02.609402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:02.609660 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:02.609826 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:02.610011 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 18:47:02.610192 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:02.621549 1475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:02.613555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.29.159.126:34625)
0927 18:47:02.613703 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:02.613707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:02.613721 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:47:02.613842 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:02.613845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:02.613884 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:02.613962 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:02.614623 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.615128 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.615973 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:02.616177 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:02.616250 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:02.618510 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:02.618513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:02.618515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:02.618538 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:02.620014 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:02.620384 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:02.620387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:02.620388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:02.620460 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:02.620692 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:02.620694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:02.620695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:02.620905 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:02.621007 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:02.621237 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 18:47:02.621389 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 18:47:02.621987 1436 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55764
I20260927 18:47:02.622607 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 1498
I20260927 18:47:02.628361 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 1632
I20260927 18:47:02.633518 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 1766
I20260927 18:47:02.638502 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 1404
2026-09-27T18:47:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2195 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1907](info): set up 2 sockets
Sep 27 18:47:02 dist-test-slave-8tg2 krb5kdc[1907](info): commencing operation
krb5kdc: starting...
W20260927 18:47:03.671452 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:47:03 dist-test-slave-8tg2 krb5kdc[1907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:47:03.788781 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:37355
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:37355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:03.898526 1920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:03.898823 1920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:03.898865 1920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:03.903054 1920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:03.903127 1920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:03.903146 1920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:03.903162 1920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:03.903177 1920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:03.908746 1920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:37355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:37355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:03.909852 1920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:03.910892 1920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:03.919186 1928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:03.919195 1926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:03.919255 1925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:03.919615 1920 server_base.cc:1034] running on GCE node
I20260927 18:47:03.920243 1920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:03.921398 1920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:03.922585 1920 hybrid_clock.cc:648] HybridClock initialized: now 1790534823922574 us; error 31 us; skew 500 ppm
Sep 27 18:47:03 dist-test-slave-8tg2 krb5kdc[1907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534823, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:03.925529 1920 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:47:03.926296 1920 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:03.926390 1920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:03.926635 1920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:03.928838 1920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "9409dcad1338422ab065f9c392dce81d"
format_stamp: "Formatted at 2026-09-27 18:47:03 on dist-test-slave-8tg2"
server_key: "8af756554b4780884fb24efedae4dc3a"
server_key_iv: "e42c9dcb5b3cab9af88c2b2dd999c7bd"
server_key_version: "encryptionkey@0"
I20260927 18:47:03.929425 1920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "9409dcad1338422ab065f9c392dce81d"
format_stamp: "Formatted at 2026-09-27 18:47:03 on dist-test-slave-8tg2"
server_key: "8af756554b4780884fb24efedae4dc3a"
server_key_iv: "e42c9dcb5b3cab9af88c2b2dd999c7bd"
server_key_version: "encryptionkey@0"
I20260927 18:47:03.933164 1920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:47:03.935578 1934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:03.936852 1920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:03.936998 1920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "9409dcad1338422ab065f9c392dce81d"
format_stamp: "Formatted at 2026-09-27 18:47:03 on dist-test-slave-8tg2"
server_key: "8af756554b4780884fb24efedae4dc3a"
server_key_iv: "e42c9dcb5b3cab9af88c2b2dd999c7bd"
server_key_version: "encryptionkey@0"
I20260927 18:47:03.937112 1920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:03.953532 1920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:03.956926 1920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:03.957131 1920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:03.964370 1920 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:37355
I20260927 18:47:03.964641 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:37355 every 8 connection(s)
I20260927 18:47:03.965106 1920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:03.968075 1987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:03.973510 1987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d: Bootstrap starting.
I20260927 18:47:03.974905 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 1920
I20260927 18:47:03.975059 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:03.975313 30333 external_mini_cluster.cc:1428] Setting key a0dd7c7f616daaa2659864d4f0cef610
I20260927 18:47:03.976166 1987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:03.976958 1987 log.cc:824] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:03.979121 1987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d: No bootstrap required, opened a new log
I20260927 18:47:03.983527 1987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9409dcad1338422ab065f9c392dce81d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37355 } }
I20260927 18:47:03.983839 1987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:03.983917 1987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9409dcad1338422ab065f9c392dce81d, State: Initialized, Role: FOLLOWER
Sep 27 18:47:03 dist-test-slave-8tg2 krb5kdc[1907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:03.984527 1987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9409dcad1338422ab065f9c392dce81d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37355 } }
I20260927 18:47:03.984716 1987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:03.984791 1987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:03.984900 1987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:03.985882 1987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9409dcad1338422ab065f9c392dce81d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37355 } }
I20260927 18:47:03.986229 1987 leader_election.cc:302] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9409dcad1338422ab065f9c392dce81d; no voters:
I20260927 18:47:03.986549 1987 leader_election.cc:288] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:03.986675 1992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:03.986908 1992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [term 1 LEADER]: Becoming Leader. State: Replica: 9409dcad1338422ab065f9c392dce81d, State: Running, Role: LEADER
I20260927 18:47:03.987282 1992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9409dcad1338422ab065f9c392dce81d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37355 } }
I20260927 18:47:03.987927 1987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:03.989496 1994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9409dcad1338422ab065f9c392dce81d. Latest consensus state: current_term: 1 leader_uuid: "9409dcad1338422ab065f9c392dce81d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9409dcad1338422ab065f9c392dce81d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37355 } } }
I20260927 18:47:03.989609 1994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:03.989882 1993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9409dcad1338422ab065f9c392dce81d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9409dcad1338422ab065f9c392dce81d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 37355 } } }
I20260927 18:47:03.990427 1993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:03.991701 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:03.976469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.29.159.126:37355)
0927 18:47:03.976936 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:03.976952 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:03.976986 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:47:03.977839 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:03.977859 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:03.978195 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:03.978482 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:03.979410 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:03.980437 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:03.981643 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:03.982099 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:03.982221 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:03.985296 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:03.985301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:03.985315 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:03.985341 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:03.987519 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:03.988011 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:03.988017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:03.988023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:03.988159 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:03.988493 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:03.988497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:03.988500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:03.989044 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:03.989286 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:03.990340 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0927 18:47:03.990679 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":147,"threads_started":1}
W20260927 18:47:03.993379 2008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:47:03.993459 2008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:47:03.993537 2001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:03.994396 2001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:03.999830 2001 catalog_manager.cc:1347] Generated new cluster ID: acb8d832b3bf4791808929a882ed9968
I20260927 18:47:03.999912 2001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:04.014415 2001 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:04.015380 2001 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:04.020082 2001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d: Generated new TSK 0
I20260927 18:47:04.020825 2001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:47:04.048785 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37355
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:04.161023 2013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:04.161286 2013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:04.161331 2013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:04.165439 2013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:04.165499 2013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:04.165560 2013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:04.170675 2013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:04.171751 2013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:04.172750 2013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:04.182297 2018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:04.182253 2019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:04.182380 2013 server_base.cc:1034] running on GCE node
W20260927 18:47:04.182267 2021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:04.182955 2013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:04.183667 2013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:04.184818 2013 hybrid_clock.cc:648] HybridClock initialized: now 1790534824184804 us; error 27 us; skew 500 ppm
Sep 27 18:47:04 dist-test-slave-8tg2 krb5kdc[1907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534824, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:04.187713 2013 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:47:04.188460 2013 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:04.188520 2013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:04.188725 2013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:04.190886 2013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "cb4201d5209543b7adf6ebd55772d063"
format_stamp: "Formatted at 2026-09-27 18:47:04 on dist-test-slave-8tg2"
server_key: "2fa0acc1c8374bcc1e4d02c93abca890"
server_key_iv: "a9f2acd326c9089dda87b0f125fb2496"
server_key_version: "encryptionkey@0"
I20260927 18:47:04.191390 2013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "cb4201d5209543b7adf6ebd55772d063"
format_stamp: "Formatted at 2026-09-27 18:47:04 on dist-test-slave-8tg2"
server_key: "2fa0acc1c8374bcc1e4d02c93abca890"
server_key_iv: "a9f2acd326c9089dda87b0f125fb2496"
server_key_version: "encryptionkey@0"
I20260927 18:47:04.194955 2013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:04.197389 2027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:04.198745 2013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:04.198844 2013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "cb4201d5209543b7adf6ebd55772d063"
format_stamp: "Formatted at 2026-09-27 18:47:04 on dist-test-slave-8tg2"
server_key: "2fa0acc1c8374bcc1e4d02c93abca890"
server_key_iv: "a9f2acd326c9089dda87b0f125fb2496"
server_key_version: "encryptionkey@0"
I20260927 18:47:04.198980 2013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:04.209976 2013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:04.213572 2013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:04.213771 2013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:04.214357 2013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:04.215210 2013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:04.215270 2013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:04.215346 2013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:04.215386 2013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:04.224481 2013 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:34507
I20260927 18:47:04.224504 2140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:34507 every 8 connection(s)
I20260927 18:47:04.225093 2013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:47:04 dist-test-slave-8tg2 krb5kdc[1907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534824, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:04.235244 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 2013
I20260927 18:47:04.235378 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:47:04.235692 30333 external_mini_cluster.cc:1428] Setting key 058a86ebe21d61e6346728e3109682ba
I20260927 18:47:04.239709 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.226890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:36377 (local address 127.29.159.126:37355)
0927 18:47:04.227040 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:04.227045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:04.228010 (+ 965us) server_negotiation.cc:407] Connection header received
0927 18:47:04.229317 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:04.229319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:04.229365 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:04.229438 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:04.230597 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.231131 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.232593 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.232820 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.232899 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.235691 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:04.235694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:04.235696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:04.235722 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:04.237383 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:04.237910 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:04.237917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:04.237921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:04.238014 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:04.238374 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:04.238380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:04.238385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:04.238635 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:04.238725 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:04.239348 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 18:47:04.239523 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:47:04.240883 2143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.227329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:37355 (local address 127.29.159.65:36377)
0927 18:47:04.227856 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:04.227899 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:04.229123 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:04.229572 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:04.229579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:04.229940 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:04.230469 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.230482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.231238 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.231240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.232481 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.232491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.232947 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.232950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.233089 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.233682 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:04.233700 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:04.235475 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:04.237509 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:04.237514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:04.237526 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:04.237790 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:04.238119 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:04.238122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:04.238124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:04.238268 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:04.238814 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:04.238822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:04.239073 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 18:47:04.239285 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 18:47:04.239682 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":125,"threads_started":1}
I20260927 18:47:04.242220 2141 heartbeater.cc:347] Connected to a master server at 127.29.159.126:37355
I20260927 18:47:04.242491 2141 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:04.243091 2141 heartbeater.cc:511] Master 127.29.159.126:37355 requested a full tablet report, sending...
I20260927 18:47:04.244781 1951 ts_manager.cc:194] Registered new tserver with Master: cb4201d5209543b7adf6ebd55772d063 (127.29.159.65:34507)
I20260927 18:47:04.245884 1951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:36377
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:47:04.264745 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37355
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:04.376724 2146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:04.377007 2146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:04.377079 2146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:04.381119 2146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:04.381201 2146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:04.381291 2146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:04.386482 2146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:04.387593 2146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:04.388592 2146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:04.398020 2152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:04.398141 2146 server_base.cc:1034] running on GCE node
W20260927 18:47:04.398078 2151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:04.398020 2154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:04.398716 2146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:04.399343 2146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:04.400520 2146 hybrid_clock.cc:648] HybridClock initialized: now 1790534824400504 us; error 27 us; skew 500 ppm
Sep 27 18:47:04 dist-test-slave-8tg2 krb5kdc[1907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534824, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:04.403481 2146 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:47:04.404211 2146 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:04.404311 2146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:04.404537 2146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:04.406507 2146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b"
format_stamp: "Formatted at 2026-09-27 18:47:04 on dist-test-slave-8tg2"
server_key: "1f452d919ebf08db319092cd1153fefa"
server_key_iv: "8cbc31f3136276ffa0c4d59350e4dc67"
server_key_version: "encryptionkey@0"
I20260927 18:47:04.407094 2146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b"
format_stamp: "Formatted at 2026-09-27 18:47:04 on dist-test-slave-8tg2"
server_key: "1f452d919ebf08db319092cd1153fefa"
server_key_iv: "8cbc31f3136276ffa0c4d59350e4dc67"
server_key_version: "encryptionkey@0"
I20260927 18:47:04.410800 2146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:04.413180 2160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:04.414647 2146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 18:47:04.414757 2146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b"
format_stamp: "Formatted at 2026-09-27 18:47:04 on dist-test-slave-8tg2"
server_key: "1f452d919ebf08db319092cd1153fefa"
server_key_iv: "8cbc31f3136276ffa0c4d59350e4dc67"
server_key_version: "encryptionkey@0"
I20260927 18:47:04.414847 2146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:04.443970 2146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:04.446902 2146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:04.447057 2146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:04.447676 2146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:04.448508 2146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:04.448555 2146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:04.448597 2146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:04.448622 2146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:04.457413 2146 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:43553
I20260927 18:47:04.457432 2273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:43553 every 8 connection(s)
I20260927 18:47:04.458031 2146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:04.461311 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 2146
I20260927 18:47:04.461449 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:04.461740 30333 external_mini_cluster.cc:1428] Setting key 356f07bbb49522f11bbab8e73b79d4d0
Sep 27 18:47:04 dist-test-slave-8tg2 krb5kdc[1907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534824, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:47:04.473317 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.459787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:59387 (local address 127.29.159.126:37355)
0927 18:47:04.459923 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:04.459926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:04.460892 (+ 966us) server_negotiation.cc:407] Connection header received
0927 18:47:04.462449 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:04.462453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:04.462509 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:04.462617 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:04.463917 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.464633 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.466077 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.466325 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.466442 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.469082 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:04.469086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:04.469088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:04.469115 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:04.470815 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:04.471549 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:04.471554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:04.471556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:04.471667 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:04.472063 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:04.472066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:04.472068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:04.472289 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:04.472409 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:04.472994 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 18:47:04.473138 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:47:04.474694 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.460239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:37355 (local address 127.29.159.66:59387)
0927 18:47:04.460733 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:04.460779 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:04.462222 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:04.462819 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:04.462827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:04.463204 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:04.463780 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.463794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.464812 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.464817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.465917 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.465927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.466487 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.466491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.466673 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.467325 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:04.467342 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:04.468886 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:04.470980 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:04.470987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:04.471006 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:04.471397 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:04.471777 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:04.471781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:04.471784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:04.471947 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:04.472421 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:04.472430 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:04.472719 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 18:47:04.472931 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 18:47:04.473442 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":112,"threads_started":1}
I20260927 18:47:04.476184 2274 heartbeater.cc:347] Connected to a master server at 127.29.159.126:37355
I20260927 18:47:04.476460 2274 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:04.476970 2274 heartbeater.cc:511] Master 127.29.159.126:37355 requested a full tablet report, sending...
I20260927 18:47:04.478128 1951 ts_manager.cc:194] Registered new tserver with Master: d2d6c18be19e4a7cac682c8c4a06fd3b (127.29.159.66:43553)
I20260927 18:47:04.478761 1951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:59387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:47:04.490906 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37355
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:04.602766 2279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:04.603034 2279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:04.603113 2279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:04.607033 2279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:04.607120 2279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:04.607224 2279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:04.612439 2279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:04.613461 2279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:04.614543 2279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:04.624150 2287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:04.624167 2284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:04.624346 2279 server_base.cc:1034] running on GCE node
W20260927 18:47:04.624218 2285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:04.624845 2279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:04.625444 2279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:04.626602 2279 hybrid_clock.cc:648] HybridClock initialized: now 1790534824626573 us; error 49 us; skew 500 ppm
Sep 27 18:47:04 dist-test-slave-8tg2 krb5kdc[1907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534824, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:04.629347 2279 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:47:04.630007 2279 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:04.630069 2279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:04.630236 2279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:04.632249 2279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "df4b23665b6a47b9b97f7ced4af8c14d"
format_stamp: "Formatted at 2026-09-27 18:47:04 on dist-test-slave-8tg2"
server_key: "2cefa2218d218cd70eebf8bb1430e5b6"
server_key_iv: "22b99dc56dd0d261459a9652784e6ec1"
server_key_version: "encryptionkey@0"
I20260927 18:47:04.632773 2279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "df4b23665b6a47b9b97f7ced4af8c14d"
format_stamp: "Formatted at 2026-09-27 18:47:04 on dist-test-slave-8tg2"
server_key: "2cefa2218d218cd70eebf8bb1430e5b6"
server_key_iv: "22b99dc56dd0d261459a9652784e6ec1"
server_key_version: "encryptionkey@0"
I20260927 18:47:04.636310 2279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:47:04.638656 2293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:04.639945 2279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:04.640038 2279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "df4b23665b6a47b9b97f7ced4af8c14d"
format_stamp: "Formatted at 2026-09-27 18:47:04 on dist-test-slave-8tg2"
server_key: "2cefa2218d218cd70eebf8bb1430e5b6"
server_key_iv: "22b99dc56dd0d261459a9652784e6ec1"
server_key_version: "encryptionkey@0"
I20260927 18:47:04.640165 2279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:04.669937 2279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:04.673004 2279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:04.673207 2279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:04.673810 2279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:04.674626 2279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:04.674674 2279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:04.674748 2279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:04.674787 2279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:04.683648 2279 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:33263
I20260927 18:47:04.683715 2406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:33263 every 8 connection(s)
I20260927 18:47:04.684321 2279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:04.687445 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 2279
I20260927 18:47:04.687582 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:04.687870 30333 external_mini_cluster.cc:1428] Setting key 06c5880ba70ba6fd24c1d2913e1acf9c
Sep 27 18:47:04 dist-test-slave-8tg2 krb5kdc[1907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534824, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:04.697526 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.685980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:53665 (local address 127.29.159.126:37355)
0927 18:47:04.686141 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:04.686145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:04.687214 (+ 1069us) server_negotiation.cc:407] Connection header received
0927 18:47:04.688253 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:04.688257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:04.688311 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:04.688399 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:04.689741 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.690367 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.691473 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.691699 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.691789 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.694168 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:04.694171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:04.694173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:04.694196 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:04.695639 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:04.696149 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:04.696152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:04.696153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:04.696227 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:04.696539 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:04.696541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:04.696543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:04.696711 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:04.696844 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:04.697300 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 18:47:04.697400 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:47:04.698392 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.686365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:37355 (local address 127.29.159.67:53665)
0927 18:47:04.687048 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:04.687088 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:04.688057 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:04.688592 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:04.688608 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:04.689039 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:04.689601 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.689615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.690477 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.690479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.691338 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.691347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.691835 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.691837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.691973 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.692576 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:04.692591 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:04.694019 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:04.695766 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:04.695772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:04.695795 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:04.696058 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:04.696315 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:04.696317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:04.696319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:04.696452 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:04.696815 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:04.696820 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:04.697063 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 18:47:04.697237 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 18:47:04.697503 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":116,"threads_started":1}
I20260927 18:47:04.699649 2407 heartbeater.cc:347] Connected to a master server at 127.29.159.126:37355
I20260927 18:47:04.699908 2407 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:04.700505 2407 heartbeater.cc:511] Master 127.29.159.126:37355 requested a full tablet report, sending...
I20260927 18:47:04.701922 1951 ts_manager.cc:194] Registered new tserver with Master: df4b23665b6a47b9b97f7ced4af8c14d (127.29.159.67:33263)
I20260927 18:47:04.702584 1951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:53665
I20260927 18:47:04.713627 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:47:04.724558 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.715785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.29.159.126:37355)
0927 18:47:04.715953 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:04.715958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:04.716139 (+ 181us) server_negotiation.cc:407] Connection header received
0927 18:47:04.716273 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:04.716276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:04.716326 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:04.716409 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:04.717186 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.717748 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.718652 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.719048 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.719157 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.720765 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:04.720770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:04.720776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:04.720809 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:04.722671 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:04.723204 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:04.723208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:04.723212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:04.723299 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:04.723636 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:04.723640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:04.723642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:04.723887 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:04.724034 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:04.724279 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 18:47:04.724396 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:47:04.727562 1951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:47:04.730523 1951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:47:04.745085 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.739010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:43553 (local address 127.0.0.1:58056)
0927 18:47:04.739860 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:04.739912 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:04.740043 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:04.740834 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:04.740839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:04.740860 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:04.741019 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.741026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.742221 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.742228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.743383 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.743392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.744502 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.744505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.744635 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.744676 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:47:04.744791 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:47:04.744907 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":103,"threads_started":1}
I20260927 18:47:04.745086 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.739147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.29.159.66:43553)
0927 18:47:04.740218 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:04.740230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:04.740254 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:47:04.740344 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:04.740352 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:04.740586 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:04.740742 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:04.741152 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.742098 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.743559 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.744170 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.744299 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.744422 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:04.744830 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 18:47:04.744934 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":97,"threads_started":1}
I20260927 18:47:04.745862 2419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.739297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:34507 (local address 127.0.0.1:48176)
0927 18:47:04.740555 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:04.740566 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:04.740704 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:04.741280 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:04.741285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:04.741305 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:04.741394 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.741399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.742903 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.742906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.744380 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.744391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.745428 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.745433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.745528 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.745555 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:47:04.745648 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:47:04.745738 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"mutex_wait_us":115,"thread_start_us":88,"threads_started":1}
I20260927 18:47:04.745899 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.739650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.29.159.65:34507)
0927 18:47:04.740001 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:04.740008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:04.740655 (+ 647us) server_negotiation.cc:407] Connection header received
0927 18:47:04.740802 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:04.740807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:04.740976 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:04.741104 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:04.741593 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.742764 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.744518 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.745330 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.745431 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.745558 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:04.745667 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 18:47:04.745761 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":73,"threads_started":1}
I20260927 18:47:04.746888 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.739020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:33263 (local address 127.0.0.1:52870)
0927 18:47:04.740283 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:04.740296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:04.740402 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:04.740997 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:04.741000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:04.741014 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:04.741108 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.741194 (+ 86us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.742903 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.742906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.744400 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.744406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.746524 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.746527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.746610 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.746641 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:47:04.746716 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:47:04.746776 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"mutex_wait_us":147,"thread_start_us":85,"threads_started":1}
I20260927 18:47:04.747020 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.739814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.29.159.67:33263)
0927 18:47:04.740234 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:04.740246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:04.740499 (+ 253us) server_negotiation.cc:407] Connection header received
0927 18:47:04.740590 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:04.740594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:04.740857 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:04.741006 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:04.741520 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.742750 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.744501 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.746249 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.746348 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.746467 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:04.746774 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 18:47:04.746878 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":105,"threads_started":1}
I20260927 18:47:04.748522 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 91f08bccfc764a1a8cf8ac70372a37e3 (DEFAULT_TABLE table=test-table [id=4ab018ce89aa4c4f9bd5f4d6964116d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:04.748625 2075 tablet_service.cc:1467] Processing CreateTablet for tablet 91f08bccfc764a1a8cf8ac70372a37e3 (DEFAULT_TABLE table=test-table [id=4ab018ce89aa4c4f9bd5f4d6964116d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:04.749543 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f08bccfc764a1a8cf8ac70372a37e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:04.749629 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f08bccfc764a1a8cf8ac70372a37e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:04.749873 2341 tablet_service.cc:1467] Processing CreateTablet for tablet 91f08bccfc764a1a8cf8ac70372a37e3 (DEFAULT_TABLE table=test-table [id=4ab018ce89aa4c4f9bd5f4d6964116d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:04.750907 2341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f08bccfc764a1a8cf8ac70372a37e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:04.755723 2424 tablet_bootstrap.cc:492] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b: Bootstrap starting.
I20260927 18:47:04.757189 2425 tablet_bootstrap.cc:492] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063: Bootstrap starting.
I20260927 18:47:04.757795 2426 tablet_bootstrap.cc:492] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: Bootstrap starting.
I20260927 18:47:04.758688 2424 tablet_bootstrap.cc:654] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:04.759100 2425 tablet_bootstrap.cc:654] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:04.759920 2425 log.cc:824] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:04.759987 2424 log.cc:824] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:04.760318 2426 tablet_bootstrap.cc:654] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:04.761536 2426 log.cc:824] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:04.761737 2424 tablet_bootstrap.cc:492] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b: No bootstrap required, opened a new log
I20260927 18:47:04.761889 2425 tablet_bootstrap.cc:492] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063: No bootstrap required, opened a new log
I20260927 18:47:04.761961 2424 ts_tablet_manager.cc:1397] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260927 18:47:04.762066 2425 ts_tablet_manager.cc:1397] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:47:04.763633 2426 tablet_bootstrap.cc:492] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: No bootstrap required, opened a new log
I20260927 18:47:04.763872 2426 ts_tablet_manager.cc:1397] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:47:04.765125 2424 raft_consensus.cc:348] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43553 } } peers { permanent_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 33263 } } peers { permanent_uuid: "cb4201d5209543b7adf6ebd55772d063" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 34507 } }
I20260927 18:47:04.765372 2424 raft_consensus.cc:374] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:04.765434 2424 raft_consensus.cc:729] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d6c18be19e4a7cac682c8c4a06fd3b, State: Initialized, Role: FOLLOWER
I20260927 18:47:04.765895 2424 consensus_queue.cc:260] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43553 } } peers { permanent_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 33263 } } peers { permanent_uuid: "cb4201d5209543b7adf6ebd55772d063" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 34507 } }
I20260927 18:47:04.766553 2274 heartbeater.cc:503] Master 127.29.159.126:37355 was elected leader, sending a full tablet report...
I20260927 18:47:04.766397 2425 raft_consensus.cc:348] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43553 } } peers { permanent_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 33263 } } peers { permanent_uuid: "cb4201d5209543b7adf6ebd55772d063" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 34507 } }
I20260927 18:47:04.766647 2425 raft_consensus.cc:374] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:04.766700 2425 raft_consensus.cc:729] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4201d5209543b7adf6ebd55772d063, State: Initialized, Role: FOLLOWER
I20260927 18:47:04.767002 2424 ts_tablet_manager.cc:1428] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260927 18:47:04.767083 2425 consensus_queue.cc:260] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43553 } } peers { permanent_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 33263 } } peers { permanent_uuid: "cb4201d5209543b7adf6ebd55772d063" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 34507 } }
I20260927 18:47:04.767773 2141 heartbeater.cc:503] Master 127.29.159.126:37355 was elected leader, sending a full tablet report...
I20260927 18:47:04.768024 2425 ts_tablet_manager.cc:1428] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 18:47:04.768803 2426 raft_consensus.cc:348] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43553 } } peers { permanent_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 33263 } } peers { permanent_uuid: "cb4201d5209543b7adf6ebd55772d063" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 34507 } }
I20260927 18:47:04.769027 2426 raft_consensus.cc:374] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:04.769081 2426 raft_consensus.cc:729] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4b23665b6a47b9b97f7ced4af8c14d, State: Initialized, Role: FOLLOWER
I20260927 18:47:04.769586 2426 consensus_queue.cc:260] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43553 } } peers { permanent_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 33263 } } peers { permanent_uuid: "cb4201d5209543b7adf6ebd55772d063" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 34507 } }
I20260927 18:47:04.770498 2426 ts_tablet_manager.cc:1428] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:47:04.770859 2407 heartbeater.cc:503] Master 127.29.159.126:37355 was elected leader, sending a full tablet report...
I20260927 18:47:04.892385 2430 raft_consensus.cc:482] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:04.892632 2430 raft_consensus.cc:504] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43553 } } peers { permanent_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 33263 } } peers { permanent_uuid: "cb4201d5209543b7adf6ebd55772d063" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 34507 } }
I20260927 18:47:04.893852 2430 leader_election.cc:288] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4b23665b6a47b9b97f7ced4af8c14d (127.29.159.67:33263), cb4201d5209543b7adf6ebd55772d063 (127.29.159.65:34507)
I20260927 18:47:04.898857 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.894144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:54267 (local address 127.29.159.67:33263)
0927 18:47:04.894283 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:04.894290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:04.894308 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:47:04.894479 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:04.894483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:04.894533 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:04.894635 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:04.894938 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.895755 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.897482 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.898115 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.898239 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.898293 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:04.898652 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 18:47:04.898726 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:47:04.898857 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.894315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:53459 (local address 127.29.159.65:34507)
0927 18:47:04.894452 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:04.894456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:04.894588 (+ 132us) server_negotiation.cc:407] Connection header received
0927 18:47:04.894692 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:04.894695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:04.894742 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:04.894818 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:04.895091 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.895815 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.897482 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.898115 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.898239 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.898290 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:04.898666 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 18:47:04.898729 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:47:04.898908 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.894041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:33263 (local address 127.29.159.66:54267)
0927 18:47:04.894239 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:04.894254 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:04.894347 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:04.894667 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:04.894670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:04.894688 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:04.894821 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.894829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.895894 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.895900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.897359 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.897365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.898285 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.898289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.898484 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.898527 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 18:47:04.898621 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 18:47:04.898744 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":235}
I20260927 18:47:04.899106 2433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.894105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:34507 (local address 127.29.159.66:53459)
0927 18:47:04.894502 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:04.894513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:04.894586 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:04.894850 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:04.894855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:04.894875 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:04.894984 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.894988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.895923 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.895926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.897353 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:04.897364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.898285 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.898289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:04.898487 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.898530 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 18:47:04.898634 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:47:04.898743 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":145,"thread_start_us":116,"threads_started":1}
I20260927 18:47:04.899735 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f08bccfc764a1a8cf8ac70372a37e3" candidate_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4201d5209543b7adf6ebd55772d063" is_pre_election: true
I20260927 18:47:04.899737 2361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f08bccfc764a1a8cf8ac70372a37e3" candidate_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" is_pre_election: true
I20260927 18:47:04.900065 2361 raft_consensus.cc:2454] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d6c18be19e4a7cac682c8c4a06fd3b in term 0.
I20260927 18:47:04.900065 2095 raft_consensus.cc:2454] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d6c18be19e4a7cac682c8c4a06fd3b in term 0.
I20260927 18:47:04.900609 2163 leader_election.cc:302] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb4201d5209543b7adf6ebd55772d063, d2d6c18be19e4a7cac682c8c4a06fd3b; no voters:
I20260927 18:47:04.900897 2430 raft_consensus.cc:2781] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:04.900996 2430 raft_consensus.cc:482] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:04.901077 2430 raft_consensus.cc:3037] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:04.902220 2430 raft_consensus.cc:504] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43553 } } peers { permanent_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 33263 } } peers { permanent_uuid: "cb4201d5209543b7adf6ebd55772d063" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 34507 } }
I20260927 18:47:04.902630 2430 leader_election.cc:288] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [CANDIDATE]: Term 1 election: Requested vote from peers df4b23665b6a47b9b97f7ced4af8c14d (127.29.159.67:33263), cb4201d5209543b7adf6ebd55772d063 (127.29.159.65:34507)
I20260927 18:47:04.902987 2361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f08bccfc764a1a8cf8ac70372a37e3" candidate_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4b23665b6a47b9b97f7ced4af8c14d"
I20260927 18:47:04.903002 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f08bccfc764a1a8cf8ac70372a37e3" candidate_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4201d5209543b7adf6ebd55772d063"
I20260927 18:47:04.903136 2361 raft_consensus.cc:3037] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:04.903136 2095 raft_consensus.cc:3037] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:04.904429 2095 raft_consensus.cc:2454] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d6c18be19e4a7cac682c8c4a06fd3b in term 1.
I20260927 18:47:04.904429 2361 raft_consensus.cc:2454] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d6c18be19e4a7cac682c8c4a06fd3b in term 1.
I20260927 18:47:04.904737 2163 leader_election.cc:302] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb4201d5209543b7adf6ebd55772d063, d2d6c18be19e4a7cac682c8c4a06fd3b; no voters:
I20260927 18:47:04.904943 2430 raft_consensus.cc:2781] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:04.905190 2430 raft_consensus.cc:686] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 1 LEADER]: Becoming Leader. State: Replica: d2d6c18be19e4a7cac682c8c4a06fd3b, State: Running, Role: LEADER
I20260927 18:47:04.905562 2430 consensus_queue.cc:237] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43553 } } peers { permanent_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 33263 } } peers { permanent_uuid: "cb4201d5209543b7adf6ebd55772d063" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 34507 } }
I20260927 18:47:04.908690 1951 catalog_manager.cc:5495] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b reported cstate change: term changed from 0 to 1, leader changed from <none> to d2d6c18be19e4a7cac682c8c4a06fd3b (127.29.159.66). New cstate: current_term: 1 leader_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d6c18be19e4a7cac682c8c4a06fd3b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 33263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb4201d5209543b7adf6ebd55772d063" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 34507 } health_report { overall_health: UNKNOWN } } }
I20260927 18:47:04.934958 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.931463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.29.159.66:43553)
0927 18:47:04.931597 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:04.931600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:04.931613 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:47:04.931769 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:04.931771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:04.931822 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:04.931908 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:47:04.932128 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.932609 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.933459 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:04.933661 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:04.933732 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:04.934002 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:47:04.934512 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:47:04.934613 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:04.934704 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 18:47:04.934750 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 18:47:04.935284 2408 tablet.cc:2369] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:04.940910 2095 raft_consensus.cc:1261] T 91f08bccfc764a1a8cf8ac70372a37e3 P cb4201d5209543b7adf6ebd55772d063 [term 1 FOLLOWER]: Refusing update from remote peer d2d6c18be19e4a7cac682c8c4a06fd3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:04.940919 2361 raft_consensus.cc:1261] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d [term 1 FOLLOWER]: Refusing update from remote peer d2d6c18be19e4a7cac682c8c4a06fd3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:04.941607 2430 consensus_queue.cc:1035] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb4201d5209543b7adf6ebd55772d063" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:04.941839 2434 consensus_queue.cc:1035] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4b23665b6a47b9b97f7ced4af8c14d" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 33263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:04.949851 2439 mvcc.cc:205] Tried to move back new op lower bound from 7334030642950762496 to 7334030642815770624. Current Snapshot: MvccSnapshot[applied={T|T < 7334030642950762496}]
I20260927 18:47:04.950469 2438 mvcc.cc:205] Tried to move back new op lower bound from 7334030642950762496 to 7334030642815770624. Current Snapshot: MvccSnapshot[applied={T|T < 7334030642950762496}]
I20260927 18:47:04.957072 1950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:04.957250 1950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:04.959868 1950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9409dcad1338422ab065f9c392dce81d: Sending DeleteTablet for 3 replicas of tablet 91f08bccfc764a1a8cf8ac70372a37e3
I20260927 18:47:04.960793 2075 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f08bccfc764a1a8cf8ac70372a37e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:04 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:48176
I20260927 18:47:04.960795 2208 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f08bccfc764a1a8cf8ac70372a37e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:04 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:58056
I20260927 18:47:04.960809 2341 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f08bccfc764a1a8cf8ac70372a37e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:04 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:52870
I20260927 18:47:04.961365 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 2013
I20260927 18:47:04.961362 2448 tablet_replica.cc:330] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: stopping tablet replica
I20260927 18:47:04.967624 2447 tablet_replica.cc:330] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b: stopping tablet replica
W20260927 18:47:04.968083 1937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:34507 (error 108)
I20260927 18:47:04.968447 2447 raft_consensus.cc:2229] T 91f08bccfc764a1a8cf8ac70372a37e3 P d2d6c18be19e4a7cac682c8c4a06fd3b [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:47:04.968641 2448 raft_consensus.cc:2229] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:04.968999 2448 raft_consensus.cc:2258] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:04.969314 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 2146
I20260927 18:47:04.969354 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.969109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:34507 (local address 127.0.0.1:48188)
0927 18:47:04.969219 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:04.969284 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:34507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 18:47:04.971094 1937 catalog_manager.cc:4584] TS cb4201d5209543b7adf6ebd55772d063 (127.29.159.65:34507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91f08bccfc764a1a8cf8ac70372a37e3: Network error: Client connection negotiation failed: client connection to 127.29.159.65:34507: connect: Connection refused (error 111)
I20260927 18:47:04.972004 2448 ts_tablet_manager.cc:1905] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:04.974560 2448 ts_tablet_manager.cc:1918] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:04.974653 2448 log.cc:1196] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/91f08bccfc764a1a8cf8ac70372a37e3
I20260927 18:47:04.974952 2448 ts_tablet_manager.cc:1939] T 91f08bccfc764a1a8cf8ac70372a37e3 P df4b23665b6a47b9b97f7ced4af8c14d: Deleting consensus metadata
I20260927 18:47:04.975744 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 2279
I20260927 18:47:04.976440 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:04.975809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:43553 (local address 127.0.0.1:58084)
0927 18:47:04.976311 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:04.976373 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.66:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428}
I20260927 18:47:04.981730 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 1920
2026-09-27T18:47:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2342 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T18:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:05Z Disabled control of system clock
I20260927 18:47:05.010792 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:39735
--webserver_interface=127.29.159.126
--webserver_port=0
--builtin_ntp_servers=127.29.159.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:39735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:47:05.121340 2454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:05.121606 2454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:05.121642 2454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:05.125699 2454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:05.125759 2454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:05.125775 2454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:05.125792 2454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:05.125808 2454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:05.130465 2454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:39735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:39735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:05.131520 2454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:05.132882 2454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:05.141124 2460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:05.141198 2462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:05.141127 2459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:05.141383 2454 server_base.cc:1034] running on GCE node
I20260927 18:47:05.142000 2454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:05.143059 2454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:05.144244 2454 hybrid_clock.cc:648] HybridClock initialized: now 1790534825144234 us; error 27 us; skew 500 ppm
I20260927 18:47:05.146054 2454 webserver.cc:460] Webserver started at http://127.29.159.126:38623/ using document root <none> and password file <none>
I20260927 18:47:05.146617 2454 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:05.146709 2454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:05.146938 2454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:05.148986 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "28e0801adfb64823a4a023309dcbf24a"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "0dbf04bda47fef895199193323a04c50"
server_key_iv: "3a2e6454144e8653a6a1a236f9d770a6"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.149536 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "28e0801adfb64823a4a023309dcbf24a"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "0dbf04bda47fef895199193323a04c50"
server_key_iv: "3a2e6454144e8653a6a1a236f9d770a6"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.153097 2454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:47:05.155485 2468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:05.156692 2454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:05.156782 2454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "28e0801adfb64823a4a023309dcbf24a"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "0dbf04bda47fef895199193323a04c50"
server_key_iv: "3a2e6454144e8653a6a1a236f9d770a6"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.156862 2454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:05.161186 2454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:05.161698 2454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:05.161825 2454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:05.168148 2454 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:39735
I20260927 18:47:05.168160 2520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:39735 every 8 connection(s)
I20260927 18:47:05.169044 2454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:05.172024 2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:05.176366 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 2454
I20260927 18:47:05.176469 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:05.176767 30333 external_mini_cluster.cc:1428] Setting key 27952e978e55c5a37bb33319098a667a
I20260927 18:47:05.177354 2521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a: Bootstrap starting.
I20260927 18:47:05.179819 2521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:05.180672 2521 log.cc:824] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:05.182740 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.178174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.29.159.126:39735)
0927 18:47:05.178830 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:05.178849 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:05.178892 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:47:05.179711 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:05.179735 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:05.180199 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:05.180576 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:05.181333 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:05.181337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:05.181350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:05.181358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:05.181407 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:05.181422 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:05.181537 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:05.181661 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 18:47:05.181911 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":91,"threads_started":1}
I20260927 18:47:05.182886 2521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a: No bootstrap required, opened a new log
I20260927 18:47:05.186590 2521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e0801adfb64823a4a023309dcbf24a" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39735 } }
I20260927 18:47:05.186831 2521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:05.186870 2521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e0801adfb64823a4a023309dcbf24a, State: Initialized, Role: FOLLOWER
I20260927 18:47:05.187253 2521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e0801adfb64823a4a023309dcbf24a" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39735 } }
I20260927 18:47:05.187376 2521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:05.187415 2521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:05.187553 2521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:05.188347 2521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e0801adfb64823a4a023309dcbf24a" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39735 } }
I20260927 18:47:05.188616 2521 leader_election.cc:302] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28e0801adfb64823a4a023309dcbf24a; no voters:
I20260927 18:47:05.188850 2521 leader_election.cc:288] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:05.189201 2526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:05.189440 2526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [term 1 LEADER]: Becoming Leader. State: Replica: 28e0801adfb64823a4a023309dcbf24a, State: Running, Role: LEADER
I20260927 18:47:05.189757 2526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e0801adfb64823a4a023309dcbf24a" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39735 } }
I20260927 18:47:05.190197 2521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:05.191308 2527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28e0801adfb64823a4a023309dcbf24a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e0801adfb64823a4a023309dcbf24a" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39735 } } }
I20260927 18:47:05.191462 2527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:05.191749 2528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28e0801adfb64823a4a023309dcbf24a. Latest consensus state: current_term: 1 leader_uuid: "28e0801adfb64823a4a023309dcbf24a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e0801adfb64823a4a023309dcbf24a" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39735 } } }
I20260927 18:47:05.191838 2528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a [sys.catalog]: This master's current role is: LEADER
W20260927 18:47:05.194720 2541 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:47:05.194784 2541 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:47:05.194864 2535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:05.195641 2535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:05.200625 2535 catalog_manager.cc:1347] Generated new cluster ID: b5b072c1a3fe425cb6815c23baa27ed3
I20260927 18:47:05.200688 2535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:05.207839 2535 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:05.208163 2535 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:05.215755 2535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a: Generated new TSK 0
I20260927 18:47:05.216360 2535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:47:05.220181 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39735
--builtin_ntp_servers=127.29.159.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:47:05.330826 2545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:05.331075 2545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:05.331121 2545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:05.334995 2545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:05.335079 2545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:05.335191 2545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:05.339850 2545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:05.340842 2545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:05.342176 2545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:05.351893 2551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:05.352090 2545 server_base.cc:1034] running on GCE node
W20260927 18:47:05.351903 2553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:05.351903 2550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:05.352577 2545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:05.353245 2545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:05.354405 2545 hybrid_clock.cc:648] HybridClock initialized: now 1790534825354394 us; error 24 us; skew 500 ppm
I20260927 18:47:05.356312 2545 webserver.cc:460] Webserver started at http://127.29.159.65:40775/ using document root <none> and password file <none>
I20260927 18:47:05.356875 2545 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:05.356969 2545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:05.357196 2545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:05.359161 2545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "22660ec2efa940339884a74b8c33c2cc"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "2f75ae9248a7bb44e5f53c28a87a4835"
server_key_iv: "0bb4cf0920410828122bac9988301a4a"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.359798 2545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "22660ec2efa940339884a74b8c33c2cc"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "2f75ae9248a7bb44e5f53c28a87a4835"
server_key_iv: "0bb4cf0920410828122bac9988301a4a"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.363498 2545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:47:05.366513 2559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:05.367929 2545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:47:05.368017 2545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "22660ec2efa940339884a74b8c33c2cc"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "2f75ae9248a7bb44e5f53c28a87a4835"
server_key_iv: "0bb4cf0920410828122bac9988301a4a"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.368115 2545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:05.372187 2545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:05.372675 2545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:05.372807 2545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:05.373245 2545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:05.374019 2545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:05.374063 2545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:05.374099 2545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:05.374121 2545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:05.382550 2545 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:33749
I20260927 18:47:05.382572 2672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:33749 every 8 connection(s)
I20260927 18:47:05.383347 2545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:05.386210 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 2545
I20260927 18:47:05.386333 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:47:05.386616 30333 external_mini_cluster.cc:1428] Setting key 055f84b8628d916ecfdf16028250621f
I20260927 18:47:05.388552 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39735
--builtin_ntp_servers=127.29.159.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 18:47:05.389317 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.385018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:34081 (local address 127.29.159.126:39735)
0927 18:47:05.385149 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:05.385156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:05.386070 (+ 914us) server_negotiation.cc:407] Connection header received
0927 18:47:05.387501 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:05.387504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:05.387536 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:05.387604 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:05.388781 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:05.388784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:05.388786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:05.388796 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:05.388812 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:05.388819 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:05.388882 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:05.389073 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 18:47:05.389122 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:47:05.390159 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.385312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39735 (local address 127.29.159.65:34081)
0927 18:47:05.385914 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:05.385971 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:05.387234 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:05.387784 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:05.387792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:05.387902 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:05.388475 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:05.388476 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:05.388513 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:05.388578 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:05.388939 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:05.388944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:05.388948 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:47:05.389064 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 18:47:05.389258 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
I20260927 18:47:05.391378 2673 heartbeater.cc:347] Connected to a master server at 127.29.159.126:39735
I20260927 18:47:05.391609 2673 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:05.392081 2673 heartbeater.cc:511] Master 127.29.159.126:39735 requested a full tablet report, sending...
I20260927 18:47:05.393543 2485 ts_manager.cc:194] Registered new tserver with Master: 22660ec2efa940339884a74b8c33c2cc (127.29.159.65:33749)
W20260927 18:47:05.497539 2676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:05.497821 2676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:05.497895 2676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:05.501915 2676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:05.502009 2676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:05.502099 2676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:05.506731 2676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:05.507790 2676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:05.509135 2676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:05.518438 2684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:05.518508 2676 server_base.cc:1034] running on GCE node
W20260927 18:47:05.518481 2681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:05.518450 2682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:05.519052 2676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:05.519697 2676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:05.520882 2676 hybrid_clock.cc:648] HybridClock initialized: now 1790534825520870 us; error 25 us; skew 500 ppm
I20260927 18:47:05.522643 2676 webserver.cc:460] Webserver started at http://127.29.159.66:38833/ using document root <none> and password file <none>
I20260927 18:47:05.523207 2676 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:05.523304 2676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:05.523582 2676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:05.525591 2676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "c87e6f75d91746f6801ccae7c236855f"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "7af4360b44c0f20b16ffd5368cedafb4"
server_key_iv: "3ef94faa45697e284c304b7d164aa389"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.526152 2676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "c87e6f75d91746f6801ccae7c236855f"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "7af4360b44c0f20b16ffd5368cedafb4"
server_key_iv: "3ef94faa45697e284c304b7d164aa389"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.529937 2676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:47:05.532415 2690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:05.533600 2676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:05.533737 2676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "c87e6f75d91746f6801ccae7c236855f"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "7af4360b44c0f20b16ffd5368cedafb4"
server_key_iv: "3ef94faa45697e284c304b7d164aa389"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.533874 2676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:05.538420 2676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:05.539011 2676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:05.539192 2676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:05.539778 2676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:05.540673 2676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:05.540745 2676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:05.540810 2676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:05.540864 2676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:05.549506 2676 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:46707
I20260927 18:47:05.549525 2803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:46707 every 8 connection(s)
I20260927 18:47:05.550347 2676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:05.555117 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 2676
I20260927 18:47:05.555240 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:05.555569 30333 external_mini_cluster.cc:1428] Setting key 50de1c216eead8213cd5ff1ca6c7859e
I20260927 18:47:05.556315 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.552156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:44795 (local address 127.29.159.126:39735)
0927 18:47:05.552296 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:05.552299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:05.553272 (+ 973us) server_negotiation.cc:407] Connection header received
0927 18:47:05.554503 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:05.554506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:05.554542 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:05.554625 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:05.555818 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:05.555821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:05.555823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:05.555837 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:05.555859 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:05.555870 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:05.555971 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:05.556121 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 18:47:05.556182 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:47:05.557292 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.552518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39735 (local address 127.29.159.66:44795)
0927 18:47:05.553151 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:05.553188 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:05.554317 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:05.554776 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:05.554783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:05.554889 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:05.555549 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:05.555550 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:05.555588 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:05.555679 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:05.555976 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:05.555987 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:05.555993 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:47:05.556171 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 18:47:05.556396 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260927 18:47:05.557765 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39735
--builtin_ntp_servers=127.29.159.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 18:47:05.558365 2804 heartbeater.cc:347] Connected to a master server at 127.29.159.126:39735
I20260927 18:47:05.558586 2804 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:05.558969 2804 heartbeater.cc:511] Master 127.29.159.126:39735 requested a full tablet report, sending...
I20260927 18:47:05.560012 2485 ts_manager.cc:194] Registered new tserver with Master: c87e6f75d91746f6801ccae7c236855f (127.29.159.66:46707)
W20260927 18:47:05.668934 2807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:05.669245 2807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:05.669320 2807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:05.673475 2807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:05.673547 2807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:05.673641 2807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:05.678323 2807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:05.679392 2807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:05.680842 2807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:05.690721 2813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:05.690920 2807 server_base.cc:1034] running on GCE node
W20260927 18:47:05.690755 2815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:05.690732 2812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:05.691532 2807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:05.692157 2807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:05.693334 2807 hybrid_clock.cc:648] HybridClock initialized: now 1790534825693314 us; error 34 us; skew 500 ppm
I20260927 18:47:05.695063 2807 webserver.cc:460] Webserver started at http://127.29.159.67:34693/ using document root <none> and password file <none>
I20260927 18:47:05.695708 2807 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:05.695803 2807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:05.696034 2807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:05.698040 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "d6d8854c9cc248ba9927ab8ab0bf1272"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "aa60c37aca5d72a758f9a1f8f5fe18f9"
server_key_iv: "089d307c269f2717282c0dd8c64089da"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.698621 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "d6d8854c9cc248ba9927ab8ab0bf1272"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "aa60c37aca5d72a758f9a1f8f5fe18f9"
server_key_iv: "089d307c269f2717282c0dd8c64089da"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.702138 2807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 18:47:05.704613 2821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:05.705888 2807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:47:05.706018 2807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "d6d8854c9cc248ba9927ab8ab0bf1272"
format_stamp: "Formatted at 2026-09-27 18:47:05 on dist-test-slave-8tg2"
server_key: "aa60c37aca5d72a758f9a1f8f5fe18f9"
server_key_iv: "089d307c269f2717282c0dd8c64089da"
server_key_version: "encryptionkey@0"
I20260927 18:47:05.706156 2807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:05.710551 2807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:05.711115 2807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:05.711297 2807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:05.711858 2807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:05.712718 2807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:05.712790 2807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:05.712853 2807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:05.712906 2807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:05.721990 2807 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:40101
I20260927 18:47:05.722012 2934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:40101 every 8 connection(s)
I20260927 18:47:05.722957 2807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:05.723769 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 2807
I20260927 18:47:05.723912 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:05.724218 30333 external_mini_cluster.cc:1428] Setting key 804ae950e077588d72d38bd2dfd432d3
I20260927 18:47:05.730187 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.725813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:53403 (local address 127.29.159.126:39735)
0927 18:47:05.725955 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:05.725959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:05.727182 (+ 1223us) server_negotiation.cc:407] Connection header received
0927 18:47:05.728553 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:05.728557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:05.728598 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:05.728671 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:05.729721 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:05.729724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:05.729727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:05.729739 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:05.729760 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:05.729770 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:05.729872 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:05.730014 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 18:47:05.730060 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:47:05.731004 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.726370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39735 (local address 127.29.159.67:53403)
0927 18:47:05.727045 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:05.727086 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:05.728355 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:05.728842 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:05.728849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:05.728939 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:05.729481 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:05.729482 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:05.729521 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:05.729585 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:05.729860 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:05.729868 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:05.729873 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:47:05.730047 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 18:47:05.730280 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
I20260927 18:47:05.732182 2935 heartbeater.cc:347] Connected to a master server at 127.29.159.126:39735
I20260927 18:47:05.732440 2935 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:05.732884 2935 heartbeater.cc:511] Master 127.29.159.126:39735 requested a full tablet report, sending...
I20260927 18:47:05.733915 2485 ts_manager.cc:194] Registered new tserver with Master: d6d8854c9cc248ba9927ab8ab0bf1272 (127.29.159.67:40101)
I20260927 18:47:05.738220 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:47:05.741840 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.740124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.29.159.126:39735)
0927 18:47:05.740236 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:05.740240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:05.740482 (+ 242us) server_negotiation.cc:407] Connection header received
0927 18:47:05.740584 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:05.740586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:05.740616 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:05.740681 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:05.741505 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:05.741507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:05.741509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:05.741519 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:05.741534 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:05.741542 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:05.741611 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:05.741700 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 18:47:05.741736 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:47:05.744537 2485 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:47:05.747596 2485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:47:05.759469 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.756205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:46707 (local address 127.0.0.1:48668)
0927 18:47:05.756646 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:05.756699 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:05.756805 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:05.757964 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:05.757972 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:05.758047 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:05.758630 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:05.758632 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:05.758652 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:05.758723 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:05.759095 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:05.759104 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:05.759110 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:47:05.759216 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 18:47:05.759294 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":90,"threads_started":1}
I20260927 18:47:05.759470 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.756341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.29.159.66:46707)
0927 18:47:05.757547 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:05.757558 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:05.757578 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:47:05.757621 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:05.757626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:05.757804 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:05.757949 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:05.758855 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:05.758861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:05.758874 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:05.758887 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:05.758936 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:05.758988 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:05.759127 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:05.759241 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 18:47:05.759311 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":114,"threads_started":1}
I20260927 18:47:05.759613 2949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.756278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:40101 (local address 127.0.0.1:39354)
0927 18:47:05.757254 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:05.757277 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:05.757354 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:05.758350 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:05.758356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:05.758377 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:05.758792 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:05.758796 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:05.758810 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:05.758842 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:05.759313 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:05.759317 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:05.759320 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:47:05.759439 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 18:47:05.759476 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":96,"threads_started":1}
I20260927 18:47:05.760078 2950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.757025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.29.159.67:40101)
0927 18:47:05.757513 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:05.757524 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:05.757549 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:47:05.757604 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:05.757610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:05.757804 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:05.757948 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:05.759090 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:05.759096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:05.759108 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:05.759120 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:05.759172 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:05.759198 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:05.759333 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:05.759870 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 18:47:05.759935 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":177,"threads_started":1}
I20260927 18:47:05.760551 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.756957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:33749 (local address 127.0.0.1:33198)
0927 18:47:05.757538 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:05.757550 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:05.757619 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:05.758849 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:05.758851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:05.758864 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:05.759822 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:05.759824 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:05.759834 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:05.759869 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:05.760322 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:05.760325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:05.760328 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:47:05.760401 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 18:47:05.760433 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":84,"threads_started":1}
I20260927 18:47:05.760915 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.757159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.29.159.65:33749)
0927 18:47:05.757578 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:05.757591 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:05.757616 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:47:05.757716 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:05.757721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:05.757866 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:05.758017 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:05.760007 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:05.760012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:05.760022 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:05.760034 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:05.760072 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:05.760100 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:05.760278 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:05.760708 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 18:47:05.760777 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":125,"threads_started":1}
I20260927 18:47:05.762609 2869 tablet_service.cc:1467] Processing CreateTablet for tablet ea5fb0e387bb4a698184e1cdb740959d (DEFAULT_TABLE table=test-table [id=7c2707c9d1064ad88477356c18c4aef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:05.762646 2738 tablet_service.cc:1467] Processing CreateTablet for tablet ea5fb0e387bb4a698184e1cdb740959d (DEFAULT_TABLE table=test-table [id=7c2707c9d1064ad88477356c18c4aef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:05.763675 2869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5fb0e387bb4a698184e1cdb740959d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:05.763689 2738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5fb0e387bb4a698184e1cdb740959d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:05.763589 2607 tablet_service.cc:1467] Processing CreateTablet for tablet ea5fb0e387bb4a698184e1cdb740959d (DEFAULT_TABLE table=test-table [id=7c2707c9d1064ad88477356c18c4aef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:05.764484 2607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5fb0e387bb4a698184e1cdb740959d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:05.769764 2952 tablet_bootstrap.cc:492] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272: Bootstrap starting.
I20260927 18:47:05.771183 2954 tablet_bootstrap.cc:492] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc: Bootstrap starting.
I20260927 18:47:05.771744 2953 tablet_bootstrap.cc:492] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: Bootstrap starting.
I20260927 18:47:05.772539 2952 tablet_bootstrap.cc:654] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:05.773397 2952 log.cc:824] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:05.773629 2953 tablet_bootstrap.cc:654] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:05.773793 2954 tablet_bootstrap.cc:654] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:05.774443 2953 log.cc:824] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:05.774556 2954 log.cc:824] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:05.775405 2952 tablet_bootstrap.cc:492] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272: No bootstrap required, opened a new log
I20260927 18:47:05.775674 2952 ts_tablet_manager.cc:1397] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 18:47:05.776153 2953 tablet_bootstrap.cc:492] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: No bootstrap required, opened a new log
I20260927 18:47:05.776368 2953 ts_tablet_manager.cc:1397] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:47:05.779186 2952 raft_consensus.cc:348] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87e6f75d91746f6801ccae7c236855f" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 46707 } } peers { permanent_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 40101 } } peers { permanent_uuid: "22660ec2efa940339884a74b8c33c2cc" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33749 } }
I20260927 18:47:05.779421 2952 raft_consensus.cc:374] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:05.779498 2952 raft_consensus.cc:729] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d8854c9cc248ba9927ab8ab0bf1272, State: Initialized, Role: FOLLOWER
I20260927 18:47:05.779634 2953 raft_consensus.cc:348] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87e6f75d91746f6801ccae7c236855f" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 46707 } } peers { permanent_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 40101 } } peers { permanent_uuid: "22660ec2efa940339884a74b8c33c2cc" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33749 } }
I20260927 18:47:05.779904 2953 raft_consensus.cc:374] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:05.779964 2953 raft_consensus.cc:729] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87e6f75d91746f6801ccae7c236855f, State: Initialized, Role: FOLLOWER
I20260927 18:47:05.779994 2952 consensus_queue.cc:260] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87e6f75d91746f6801ccae7c236855f" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 46707 } } peers { permanent_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 40101 } } peers { permanent_uuid: "22660ec2efa940339884a74b8c33c2cc" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33749 } }
I20260927 18:47:05.780411 2953 consensus_queue.cc:260] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87e6f75d91746f6801ccae7c236855f" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 46707 } } peers { permanent_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 40101 } } peers { permanent_uuid: "22660ec2efa940339884a74b8c33c2cc" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33749 } }
I20260927 18:47:05.780707 2935 heartbeater.cc:503] Master 127.29.159.126:39735 was elected leader, sending a full tablet report...
I20260927 18:47:05.781018 2952 ts_tablet_manager.cc:1428] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260927 18:47:05.781163 2954 tablet_bootstrap.cc:492] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc: No bootstrap required, opened a new log
I20260927 18:47:05.781344 2954 ts_tablet_manager.cc:1397] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260927 18:47:05.781401 2804 heartbeater.cc:503] Master 127.29.159.126:39735 was elected leader, sending a full tablet report...
I20260927 18:47:05.781992 2953 ts_tablet_manager.cc:1428] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:47:05.785730 2954 raft_consensus.cc:348] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87e6f75d91746f6801ccae7c236855f" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 46707 } } peers { permanent_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 40101 } } peers { permanent_uuid: "22660ec2efa940339884a74b8c33c2cc" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33749 } }
I20260927 18:47:05.785992 2954 raft_consensus.cc:374] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:05.786041 2954 raft_consensus.cc:729] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22660ec2efa940339884a74b8c33c2cc, State: Initialized, Role: FOLLOWER
I20260927 18:47:05.786402 2954 consensus_queue.cc:260] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87e6f75d91746f6801ccae7c236855f" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 46707 } } peers { permanent_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 40101 } } peers { permanent_uuid: "22660ec2efa940339884a74b8c33c2cc" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33749 } }
I20260927 18:47:05.786967 2673 heartbeater.cc:503] Master 127.29.159.126:39735 was elected leader, sending a full tablet report...
I20260927 18:47:05.787146 2958 raft_consensus.cc:482] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:05.787207 2954 ts_tablet_manager.cc:1428] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 18:47:05.787305 2958 raft_consensus.cc:504] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87e6f75d91746f6801ccae7c236855f" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 46707 } } peers { permanent_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 40101 } } peers { permanent_uuid: "22660ec2efa940339884a74b8c33c2cc" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33749 } }
I20260927 18:47:05.788357 2958 leader_election.cc:288] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87e6f75d91746f6801ccae7c236855f (127.29.159.66:46707), 22660ec2efa940339884a74b8c33c2cc (127.29.159.65:33749)
I20260927 18:47:05.790565 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.788524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:46707 (local address 127.29.159.67:45103)
0927 18:47:05.788648 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:05.788663 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:05.788730 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:05.789114 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:05.789117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:05.789134 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:05.789664 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:05.789665 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:05.789681 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:05.789734 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:05.790012 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:05.790018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:05.790025 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:47:05.790375 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 18:47:05.790413 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 18:47:05.790904 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.788721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:45103 (local address 127.29.159.66:46707)
0927 18:47:05.788824 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:05.788828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:05.788841 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:47:05.788877 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:05.788880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:05.788909 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:05.788988 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:05.789868 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:05.789872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:05.789875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:05.789888 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:05.789909 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:05.789919 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:05.790011 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:05.790726 (+ 715us) server_negotiation.cc:299] Negotiation successful
0927 18:47:05.790763 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:47:05.791077 2951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.789169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:51791 (local address 127.29.159.65:33749)
0927 18:47:05.789278 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:05.789283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:05.789551 (+ 268us) server_negotiation.cc:407] Connection header received
0927 18:47:05.789652 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:05.789655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:05.789695 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:05.789772 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:05.790508 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:05.790537 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:05.790540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:05.790558 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:05.790579 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:05.790591 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:05.790682 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:05.790917 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 18:47:05.790958 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:47:05.791112 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.788788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:33749 (local address 127.29.159.67:51791)
0927 18:47:05.789449 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:05.789464 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:05.789547 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:05.789867 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:05.789871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:05.789887 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:05.790351 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:05.790353 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:05.790362 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:05.790390 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:05.790830 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:05.790833 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:05.790836 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 18:47:05.790908 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:47:05.790935 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":124,"threads_started":1}
I20260927 18:47:05.791458 2758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5fb0e387bb4a698184e1cdb740959d" candidate_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87e6f75d91746f6801ccae7c236855f" is_pre_election: true
I20260927 18:47:05.791734 2758 raft_consensus.cc:2454] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6d8854c9cc248ba9927ab8ab0bf1272 in term 0.
I20260927 18:47:05.792301 2824 leader_election.cc:302] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c87e6f75d91746f6801ccae7c236855f, d6d8854c9cc248ba9927ab8ab0bf1272; no voters:
I20260927 18:47:05.792433 2627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5fb0e387bb4a698184e1cdb740959d" candidate_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22660ec2efa940339884a74b8c33c2cc" is_pre_election: true
I20260927 18:47:05.792568 2958 raft_consensus.cc:2781] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:05.792644 2958 raft_consensus.cc:482] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:05.792665 2627 raft_consensus.cc:2454] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6d8854c9cc248ba9927ab8ab0bf1272 in term 0.
I20260927 18:47:05.792708 2958 raft_consensus.cc:3037] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:05.793749 2958 raft_consensus.cc:504] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87e6f75d91746f6801ccae7c236855f" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 46707 } } peers { permanent_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 40101 } } peers { permanent_uuid: "22660ec2efa940339884a74b8c33c2cc" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33749 } }
I20260927 18:47:05.794039 2958 leader_election.cc:288] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [CANDIDATE]: Term 1 election: Requested vote from peers c87e6f75d91746f6801ccae7c236855f (127.29.159.66:46707), 22660ec2efa940339884a74b8c33c2cc (127.29.159.65:33749)
I20260927 18:47:05.794337 2758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5fb0e387bb4a698184e1cdb740959d" candidate_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87e6f75d91746f6801ccae7c236855f"
I20260927 18:47:05.794361 2627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5fb0e387bb4a698184e1cdb740959d" candidate_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22660ec2efa940339884a74b8c33c2cc"
I20260927 18:47:05.794430 2758 raft_consensus.cc:3037] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:05.794477 2627 raft_consensus.cc:3037] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:05.795341 2758 raft_consensus.cc:2454] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6d8854c9cc248ba9927ab8ab0bf1272 in term 1.
I20260927 18:47:05.795496 2627 raft_consensus.cc:2454] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6d8854c9cc248ba9927ab8ab0bf1272 in term 1.
I20260927 18:47:05.795616 2824 leader_election.cc:302] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c87e6f75d91746f6801ccae7c236855f, d6d8854c9cc248ba9927ab8ab0bf1272; no voters:
I20260927 18:47:05.795815 2958 raft_consensus.cc:2781] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:05.796022 2958 raft_consensus.cc:686] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 1 LEADER]: Becoming Leader. State: Replica: d6d8854c9cc248ba9927ab8ab0bf1272, State: Running, Role: LEADER
I20260927 18:47:05.796305 2958 consensus_queue.cc:237] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87e6f75d91746f6801ccae7c236855f" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 46707 } } peers { permanent_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 40101 } } peers { permanent_uuid: "22660ec2efa940339884a74b8c33c2cc" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33749 } }
I20260927 18:47:05.799351 2485 catalog_manager.cc:5495] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6d8854c9cc248ba9927ab8ab0bf1272 (127.29.159.67). New cstate: current_term: 1 leader_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87e6f75d91746f6801ccae7c236855f" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 46707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6d8854c9cc248ba9927ab8ab0bf1272" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 40101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22660ec2efa940339884a74b8c33c2cc" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33749 } health_report { overall_health: UNKNOWN } } }
W20260927 18:47:05.801326 2805 tablet.cc:2369] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:05.813830 2950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.812495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.29.159.67:40101)
0927 18:47:05.812615 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:05.812619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:05.812631 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:47:05.812765 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:05.812768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:05.812819 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:05.812889 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:05.813511 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:05.813514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:05.813516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:05.813526 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:05.813544 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:05.813552 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:05.813621 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:05.813705 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 18:47:05.813735 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:47:05.819504 2627 raft_consensus.cc:1261] T ea5fb0e387bb4a698184e1cdb740959d P 22660ec2efa940339884a74b8c33c2cc [term 1 FOLLOWER]: Refusing update from remote peer d6d8854c9cc248ba9927ab8ab0bf1272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:05.819536 2758 raft_consensus.cc:1261] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f [term 1 FOLLOWER]: Refusing update from remote peer d6d8854c9cc248ba9927ab8ab0bf1272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:05.820210 2962 consensus_queue.cc:1035] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22660ec2efa940339884a74b8c33c2cc" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:05.820428 2958 consensus_queue.cc:1035] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87e6f75d91746f6801ccae7c236855f" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 46707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:05.835975 2483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:05.836114 2483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:05.838527 2483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28e0801adfb64823a4a023309dcbf24a: Sending DeleteTablet for 3 replicas of tablet ea5fb0e387bb4a698184e1cdb740959d
I20260927 18:47:05.839226 2869 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5fb0e387bb4a698184e1cdb740959d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:05 UTC) from {username='slave'} at 127.0.0.1:39354
I20260927 18:47:05.839378 2607 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5fb0e387bb4a698184e1cdb740959d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:05 UTC) from {username='slave'} at 127.0.0.1:33198
I20260927 18:47:05.839447 2738 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5fb0e387bb4a698184e1cdb740959d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:05 UTC) from {username='slave'} at 127.0.0.1:48668
I20260927 18:47:05.839798 2975 tablet_replica.cc:330] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272: stopping tablet replica
I20260927 18:47:05.840063 2977 tablet_replica.cc:330] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: stopping tablet replica
I20260927 18:47:05.840363 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 2545
I20260927 18:47:05.840389 2977 raft_consensus.cc:2229] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:05.840095 2975 raft_consensus.cc:2229] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:47:05.842547 2977 raft_consensus.cc:2258] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:05.842787 2975 raft_consensus.cc:2258] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:05.844166 2975 ts_tablet_manager.cc:1905] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:05.844300 2977 ts_tablet_manager.cc:1905] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:05.847004 2975 ts_tablet_manager.cc:1918] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:05.847175 2975 log.cc:1196] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/ea5fb0e387bb4a698184e1cdb740959d
I20260927 18:47:05.847609 2975 ts_tablet_manager.cc:1939] T ea5fb0e387bb4a698184e1cdb740959d P d6d8854c9cc248ba9927ab8ab0bf1272: Deleting consensus metadata
I20260927 18:47:05.847656 2977 ts_tablet_manager.cc:1918] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:05.847760 2977 log.cc:1196] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/ea5fb0e387bb4a698184e1cdb740959d
I20260927 18:47:05.848088 2977 ts_tablet_manager.cc:1939] T ea5fb0e387bb4a698184e1cdb740959d P c87e6f75d91746f6801ccae7c236855f: Deleting consensus metadata
W20260927 18:47:05.848412 2471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:33749 (error 108)
I20260927 18:47:05.849017 2469 catalog_manager.cc:4841] TS d6d8854c9cc248ba9927ab8ab0bf1272 (127.29.159.67:40101): tablet ea5fb0e387bb4a698184e1cdb740959d (table test-table [id=7c2707c9d1064ad88477356c18c4aef7]) successfully deleted
I20260927 18:47:05.849170 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 2676
I20260927 18:47:05.849217 2471 catalog_manager.cc:4841] TS c87e6f75d91746f6801ccae7c236855f (127.29.159.66:46707): tablet ea5fb0e387bb4a698184e1cdb740959d (table test-table [id=7c2707c9d1064ad88477356c18c4aef7]) successfully deleted
I20260927 18:47:05.850835 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:05.850561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:33749 (local address 127.0.0.1:33210)
0927 18:47:05.850686 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:05.850755 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:33749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 18:47:05.850996 2471 catalog_manager.cc:4584] TS 22660ec2efa940339884a74b8c33c2cc (127.29.159.65:33749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea5fb0e387bb4a698184e1cdb740959d: Network error: Client connection negotiation failed: client connection to 127.29.159.65:33749: connect: Connection refused (error 111)
I20260927 18:47:05.854578 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 2807
I20260927 18:47:05.860240 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 2454
2026-09-27T18:47:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (878 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 18:47:05.881353 30333 mini_oidc.cc:152] Starting JWKS server
I20260927 18:47:05.882848 30333 webserver.cc:460] Webserver started at https://127.0.0.1:35771/ using document root <none> and password file <none>
I20260927 18:47:05.882980 30333 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:47:05.883198 30333 webserver.cc:460] Webserver started at http://127.0.0.1:45331/ using document root <none> and password file <none>
2026-09-27T18:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:05Z Disabled control of system clock
I20260927 18:47:05.891633 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:46871
--webserver_interface=127.29.159.126
--webserver_port=0
--builtin_ntp_servers=127.29.159.84:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:46871
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35771/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/testchainca.cert with env {}
W20260927 18:47:06.000473 2984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:06.000763 2984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:06.000846 2984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:06.004840 2984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:06.004948 2984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:06.004987 2984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:06.005024 2984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:06.005056 2984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:47:06.005115 2984 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:47:06.005144 2984 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35771/jwks.json
I20260927 18:47:06.010638 2984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:46871
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:46871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35771/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:06.011762 2984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:06.012812 2984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:06.021188 2990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:06.021363 2984 server_base.cc:1034] running on GCE node
W20260927 18:47:06.021195 2989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:06.021195 2992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:06.022048 2984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:06.023108 2984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:06.024278 2984 hybrid_clock.cc:648] HybridClock initialized: now 1790534826024241 us; error 55 us; skew 500 ppm
I20260927 18:47:06.026145 2984 webserver.cc:460] Webserver started at http://127.29.159.126:35179/ using document root <none> and password file <none>
I20260927 18:47:06.026723 2984 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:06.026785 2984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:06.027001 2984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:06.029012 2984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "51ff0decc324401886e659c019a2f28d"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "3948976a90d98d1230660b05f1900000"
server_key_iv: "124e2932728798deb4b7cc4b6762bd81"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.029562 2984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "51ff0decc324401886e659c019a2f28d"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "3948976a90d98d1230660b05f1900000"
server_key_iv: "124e2932728798deb4b7cc4b6762bd81"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.033058 2984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:47:06.035385 2998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:06.036743 2984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:06.036863 2984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "51ff0decc324401886e659c019a2f28d"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "3948976a90d98d1230660b05f1900000"
server_key_iv: "124e2932728798deb4b7cc4b6762bd81"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.036965 2984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:06.053673 2984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:06.054352 2984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:06.054543 2984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:06.061683 2984 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:46871
I20260927 18:47:06.061741 3050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:46871 every 8 connection(s)
I20260927 18:47:06.062636 2984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:06.065724 3051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:06.066917 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 2984
I20260927 18:47:06.067018 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:06.067361 30333 external_mini_cluster.cc:1428] Setting key 1362bd40baf3a7381a4c212fdbba2a2a
I20260927 18:47:06.071806 3051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d: Bootstrap starting.
I20260927 18:47:06.074026 3051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:06.074784 3051 log.cc:824] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:06.076439 3054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:06.068754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.29.159.126:46871)
0927 18:47:06.069406 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:06.069422 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:06.069456 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:47:06.070087 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:06.070102 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:06.070505 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:06.070846 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:06.071273 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.072341 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.073217 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.073691 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.073868 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:06.074707 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:06.074712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:06.074727 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:06.075156 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:06.075209 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:06.075229 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:06.075343 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:06.075505 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 18:47:06.075752 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":139,"threads_started":1}
I20260927 18:47:06.076831 3051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d: No bootstrap required, opened a new log
I20260927 18:47:06.079223 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:47:06.080739 3051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff0decc324401886e659c019a2f28d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46871 } }
I20260927 18:47:06.081080 3051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:06.081163 3051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ff0decc324401886e659c019a2f28d, State: Initialized, Role: FOLLOWER
I20260927 18:47:06.081593 3051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff0decc324401886e659c019a2f28d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46871 } }
I20260927 18:47:06.081751 3051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:06.081840 3051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:06.081939 3051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:06.082859 3051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff0decc324401886e659c019a2f28d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46871 } }
I20260927 18:47:06.083223 3051 leader_election.cc:302] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51ff0decc324401886e659c019a2f28d; no voters:
I20260927 18:47:06.083633 3051 leader_election.cc:288] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:06.083755 3056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:06.083937 3056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [term 1 LEADER]: Becoming Leader. State: Replica: 51ff0decc324401886e659c019a2f28d, State: Running, Role: LEADER
I20260927 18:47:06.084237 3056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff0decc324401886e659c019a2f28d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46871 } }
I20260927 18:47:06.084930 3051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:06.086069 3057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51ff0decc324401886e659c019a2f28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff0decc324401886e659c019a2f28d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46871 } } }
I20260927 18:47:06.086193 3057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:06.086483 3058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51ff0decc324401886e659c019a2f28d. Latest consensus state: current_term: 1 leader_uuid: "51ff0decc324401886e659c019a2f28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff0decc324401886e659c019a2f28d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46871 } } }
I20260927 18:47:06.086596 3058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:06.086609 3065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:06.089521 3065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:06.095588 3065 catalog_manager.cc:1347] Generated new cluster ID: 04442bd282314b37b6edff1438339a2d
I20260927 18:47:06.095682 3065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:06.107513 3065 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:06.108495 3065 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:06.114143 3065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51ff0decc324401886e659c019a2f28d: Generated new TSK 0
I20260927 18:47:06.114995 3065 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:47:06.148218 3081 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 18:47:06.148370 3081 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:47:06.149021 3054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:06.130825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.29.159.126:46871)
0927 18:47:06.130970 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:06.130975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:06.131176 (+ 201us) server_negotiation.cc:407] Connection header received
0927 18:47:06.131361 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:06.131365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:06.131446 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:06.131569 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:47:06.131805 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.132693 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.133605 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.133991 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.134099 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:06.134505 (+ 406us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:47:06.148531 (+ 14026us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:47:06.148854 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 18:47:06.149124 3054 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 18:47:06.150434 3080 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:06.130697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46871 (local address 127.0.0.1:34922)
0927 18:47:06.131047 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:06.131060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:06.131180 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:06.131575 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:06.131577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:06.131584 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:47:06.131694 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:06.131699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.132822 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.132825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:06.133481 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:06.133490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.134133 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.134138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:06.134226 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:06.134279 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:47:06.148851 (+ 14572us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:47:06.149254 (+ 403us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.159.126:46871: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":137,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.29.159.126:46871: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 18:47:06.151409 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 2984
2026-09-27T18:47:06Z chronyd exiting
I20260927 18:47:06.189185 30333 test_util.cc:175] -----------------------------------------------
I20260927 18:47:06.189306 30333 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534801142164-30333-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 18:47:06.190353 30333 mini_oidc.cc:152] Starting JWKS server
I20260927 18:47:06.191220 30333 webserver.cc:460] Webserver started at https://127.0.0.1:34951/ using document root <none> and password file <none>
I20260927 18:47:06.191358 30333 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:47:06.191700 30333 webserver.cc:460] Webserver started at http://127.0.0.1:36115/ using document root <none> and password file <none>
2026-09-27T18:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:06Z Disabled control of system clock
I20260927 18:47:06.200716 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:36809
--webserver_interface=127.29.159.126
--webserver_port=0
--builtin_ntp_servers=127.29.159.84:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:36809
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34951/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/testchainca.cert with env {}
W20260927 18:47:06.310467 3087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:06.310774 3087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:06.310817 3087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:06.315021 3087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:06.315096 3087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:06.315114 3087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:06.315130 3087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:06.315146 3087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:47:06.315207 3087 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:47:06.315225 3087 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34951/jwks.json
I20260927 18:47:06.321031 3087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:36809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:36809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34951/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:06.322201 3087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:06.323288 3087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:06.331961 3092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:06.332015 3087 server_base.cc:1034] running on GCE node
W20260927 18:47:06.331974 3095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:06.332042 3093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:06.332788 3087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:06.333868 3087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:06.335058 3087 hybrid_clock.cc:648] HybridClock initialized: now 1790534826335037 us; error 34 us; skew 500 ppm
I20260927 18:47:06.336880 3087 webserver.cc:460] Webserver started at http://127.29.159.126:33379/ using document root <none> and password file <none>
I20260927 18:47:06.337448 3087 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:06.337538 3087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:06.337783 3087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:06.339805 3087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "4654c7e8145744dca2deb07237ce35f9"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "181adbbb4b895faa8c28b09f1522a48d"
server_key_iv: "a5ea019cff94ace514d18ecec34d4671"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.340348 3087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "4654c7e8145744dca2deb07237ce35f9"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "181adbbb4b895faa8c28b09f1522a48d"
server_key_iv: "a5ea019cff94ace514d18ecec34d4671"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.343915 3087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:47:06.346225 3101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:06.347527 3087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:06.347661 3087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "4654c7e8145744dca2deb07237ce35f9"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "181adbbb4b895faa8c28b09f1522a48d"
server_key_iv: "a5ea019cff94ace514d18ecec34d4671"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.347781 3087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:06.363466 3087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:06.364128 3087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:06.364331 3087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:06.371968 3087 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:36809
I20260927 18:47:06.371986 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:36809 every 8 connection(s)
I20260927 18:47:06.372895 3087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:06.376274 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 3087
I20260927 18:47:06.376296 3154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:06.376417 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:06.376703 30333 external_mini_cluster.cc:1428] Setting key 3230f19161a37580a6029ab53f088ea7
I20260927 18:47:06.383224 3154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9: Bootstrap starting.
I20260927 18:47:06.385349 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:06.377973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.29.159.126:36809)
0927 18:47:06.378564 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:06.378578 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:06.378611 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:47:06.379250 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:06.379268 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:06.379700 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:06.380032 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:06.380445 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.381383 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.382243 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.382699 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.382886 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:06.383762 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:06.383768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:06.383779 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:06.384146 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:06.384198 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:06.384217 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:06.384330 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:06.384444 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 18:47:06.384685 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260927 18:47:06.385530 3154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:06.386415 3154 log.cc:824] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:06.388165 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:47:06.388557 3154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9: No bootstrap required, opened a new log
I20260927 18:47:06.392227 3154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4654c7e8145744dca2deb07237ce35f9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36809 } }
I20260927 18:47:06.392424 3154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:06.392508 3154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4654c7e8145744dca2deb07237ce35f9, State: Initialized, Role: FOLLOWER
I20260927 18:47:06.392907 3154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4654c7e8145744dca2deb07237ce35f9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36809 } }
I20260927 18:47:06.393060 3154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:06.393141 3154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:06.393261 3154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:06.394210 3154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4654c7e8145744dca2deb07237ce35f9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36809 } }
I20260927 18:47:06.394483 3154 leader_election.cc:302] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4654c7e8145744dca2deb07237ce35f9; no voters:
I20260927 18:47:06.394726 3154 leader_election.cc:288] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:06.394834 3159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:06.395098 3159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [term 1 LEADER]: Becoming Leader. State: Replica: 4654c7e8145744dca2deb07237ce35f9, State: Running, Role: LEADER
I20260927 18:47:06.395385 3159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4654c7e8145744dca2deb07237ce35f9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36809 } }
I20260927 18:47:06.395895 3154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:06.397037 3161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4654c7e8145744dca2deb07237ce35f9. Latest consensus state: current_term: 1 leader_uuid: "4654c7e8145744dca2deb07237ce35f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4654c7e8145744dca2deb07237ce35f9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36809 } } }
I20260927 18:47:06.397156 3161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:06.397404 3160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4654c7e8145744dca2deb07237ce35f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4654c7e8145744dca2deb07237ce35f9" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36809 } } }
I20260927 18:47:06.397495 3160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:06.397643 3168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:06.400722 3168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:06.405977 3168 catalog_manager.cc:1347] Generated new cluster ID: 4b855d45d019425e960215d006176cef
I20260927 18:47:06.406034 3168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:06.421380 3168 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:06.422271 3168 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:06.427825 3168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4654c7e8145744dca2deb07237ce35f9: Generated new TSK 0
I20260927 18:47:06.428583 3168 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:47:06.461529 3184 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 18:47:06.461624 3184 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:47:06.462248 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:06.438556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.29.159.126:36809)
0927 18:47:06.438704 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:06.438708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:06.438886 (+ 178us) server_negotiation.cc:407] Connection header received
0927 18:47:06.439038 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:06.439041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:06.439084 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:06.439173 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:47:06.439419 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.439985 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.441220 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.441433 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.441521 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:06.441844 (+ 323us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:47:06.461897 (+ 20053us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:47:06.462134 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41016: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 18:47:06.462332 3157 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41016: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 18:47:06.462476 3183 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:06.438484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36809 (local address 127.0.0.1:41016)
0927 18:47:06.438797 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:06.438812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:06.438923 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:06.439199 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:06.439202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:06.439209 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:47:06.439321 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:06.439327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.440100 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.440104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:06.441084 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:06.441095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.441568 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.441572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:06.441661 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:06.441688 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:47:06.462103 (+ 20415us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:47:06.462296 (+ 193us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.159.126:36809: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
I20260927 18:47:06.463296 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 3087
2026-09-27T18:47:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 18:47:06.501766 30333 mini_oidc.cc:152] Starting JWKS server
I20260927 18:47:06.502651 30333 webserver.cc:460] Webserver started at https://127.0.0.1:45905/ using document root <none> and password file <none>
I20260927 18:47:06.502767 30333 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:47:06.503094 30333 webserver.cc:460] Webserver started at http://127.0.0.1:37737/ using document root <none> and password file <none>
2026-09-27T18:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:06Z Disabled control of system clock
I20260927 18:47:06.511587 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:38909
--webserver_interface=127.29.159.126
--webserver_port=0
--builtin_ntp_servers=127.29.159.84:37191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:38909
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45905/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:47:06.623018 3190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:06.623291 3190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:06.623329 3190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:06.627336 3190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:06.627389 3190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:06.627405 3190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:06.627435 3190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:06.627451 3190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:47:06.627519 3190 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:47:06.627535 3190 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45905/jwks.json
I20260927 18:47:06.632222 3190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:37191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:38909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:38909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45905/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:06.633237 3190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:06.634594 3190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:47:06.643095 3190 server_base.cc:1034] running on GCE node
W20260927 18:47:06.643035 3198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:06.643046 3195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:06.643035 3196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:06.643867 3190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:06.644918 3190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:06.646072 3190 hybrid_clock.cc:648] HybridClock initialized: now 1790534826646055 us; error 34 us; skew 500 ppm
I20260927 18:47:06.647892 3190 webserver.cc:460] Webserver started at http://127.29.159.126:42653/ using document root <none> and password file <none>
I20260927 18:47:06.648494 3190 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:06.648552 3190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:06.648788 3190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:06.650731 3190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "c0e5a021356c45e1a6abf7e005625e52"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "ac3ab01b48baf76d499a267bb7c4eeda"
server_key_iv: "19da302ba90d2dadbda68d6c4e40a6c6"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.651288 3190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "c0e5a021356c45e1a6abf7e005625e52"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "ac3ab01b48baf76d499a267bb7c4eeda"
server_key_iv: "19da302ba90d2dadbda68d6c4e40a6c6"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.654850 3190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:06.657290 3204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:06.658519 3190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:06.658690 3190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "c0e5a021356c45e1a6abf7e005625e52"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "ac3ab01b48baf76d499a267bb7c4eeda"
server_key_iv: "19da302ba90d2dadbda68d6c4e40a6c6"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.658808 3190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:06.679329 3190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:06.680130 3190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:06.680343 3190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:06.687170 3190 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:38909
I20260927 18:47:06.687255 3256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:38909 every 8 connection(s)
I20260927 18:47:06.688167 3190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:06.691215 3257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:06.696583 3257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52: Bootstrap starting.
I20260927 18:47:06.697152 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 3190
I20260927 18:47:06.697297 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:06.697570 30333 external_mini_cluster.cc:1428] Setting key 86109a316290dd4763b00c519deec4f0
I20260927 18:47:06.699326 3257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:06.700227 3257 log.cc:824] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:06.702404 3257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52: No bootstrap required, opened a new log
I20260927 18:47:06.706777 3257 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e5a021356c45e1a6abf7e005625e52" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38909 } }
I20260927 18:47:06.707041 3257 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:06.707098 3257 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e5a021356c45e1a6abf7e005625e52, State: Initialized, Role: FOLLOWER
I20260927 18:47:06.707580 3257 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e5a021356c45e1a6abf7e005625e52" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38909 } }
I20260927 18:47:06.707753 3257 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:06.707823 3257 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:06.707916 3257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:06.707999 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:06.698879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.29.159.126:38909)
0927 18:47:06.699634 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:06.699653 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:06.699698 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:47:06.700554 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:06.700580 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:06.700932 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:06.701318 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:06.701736 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.703221 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.704225 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.704947 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.705103 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:06.705929 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:06.705939 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:06.705958 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:06.706461 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:06.706528 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:06.706553 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:06.706653 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:06.706749 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 18:47:06.707016 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":110,"threads_started":1}
I20260927 18:47:06.708817 3257 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e5a021356c45e1a6abf7e005625e52" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38909 } }
I20260927 18:47:06.709156 3257 leader_election.cc:302] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0e5a021356c45e1a6abf7e005625e52; no voters:
I20260927 18:47:06.709474 3257 leader_election.cc:288] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:06.709545 3262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:06.709728 3262 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [term 1 LEADER]: Becoming Leader. State: Replica: c0e5a021356c45e1a6abf7e005625e52, State: Running, Role: LEADER
I20260927 18:47:06.709995 3262 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e5a021356c45e1a6abf7e005625e52" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38909 } }
I20260927 18:47:06.710619 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:47:06.710700 3257 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:06.712148 3263 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0e5a021356c45e1a6abf7e005625e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e5a021356c45e1a6abf7e005625e52" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38909 } } }
I20260927 18:47:06.712277 3263 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:06.712710 3264 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0e5a021356c45e1a6abf7e005625e52. Latest consensus state: current_term: 1 leader_uuid: "c0e5a021356c45e1a6abf7e005625e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e5a021356c45e1a6abf7e005625e52" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38909 } } }
I20260927 18:47:06.713102 3264 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52 [sys.catalog]: This master's current role is: LEADER
W20260927 18:47:06.715051 3276 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:47:06.715150 3276 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:47:06.715248 3277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:06.716193 3277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:06.723659 3277 catalog_manager.cc:1347] Generated new cluster ID: 4a9225a80a1c4855ae585f1f4d2af2e2
I20260927 18:47:06.723760 3277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:06.740423 3277 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:06.741572 3277 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:06.753031 3277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0e5a021356c45e1a6abf7e005625e52: Generated new TSK 0
I20260927 18:47:06.753813 3277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:47:06.775748 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:06.770566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.29.159.126:38909)
0927 18:47:06.770702 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:06.770707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:06.770869 (+ 162us) server_negotiation.cc:407] Connection header received
0927 18:47:06.771028 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:06.771031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:06.771088 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:06.771206 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:47:06.771417 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.772252 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.773153 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.773498 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.773598 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:06.773863 (+ 265us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:47:06.775336 (+ 1473us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45905/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:47:06.775613 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45905/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 18:47:06.775820 3260 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45905/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 18:47:06.777387 3286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:06.770337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:38909 (local address 127.0.0.1:56894)
0927 18:47:06.770783 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:06.770795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:06.770891 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:06.771211 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:06.771213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:06.771220 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:47:06.771314 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:06.771318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.772376 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.772378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:06.773046 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:06.773053 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:06.773608 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:06.773611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:06.773679 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:06.773700 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:47:06.775526 (+ 1826us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45905/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:47:06.777223 (+ 1697us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.159.126:38909: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45905/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1520,"thread_start_us":125,"threads_started":1}
I20260927 18:47:06.778352 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 3190
E20260927 18:47:06.786187 3287 webserver.cc:533] Webserver: sslize failed:
E20260927 18:47:06.786247 3287 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T18:47:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T18:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:06Z Disabled control of system clock
I20260927 18:47:06.825898 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:32835
--webserver_interface=127.29.159.126
--webserver_port=0
--builtin_ntp_servers=127.29.159.84:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:32835
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/testchainca.cert with env {}
W20260927 18:47:06.933386 3291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:06.933849 3291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:06.933956 3291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:06.941077 3291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:06.941162 3291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:06.941179 3291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:06.941232 3291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:06.941263 3291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:47:06.941295 3291 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:47:06.941336 3291 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 18:47:06.946857 3291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:32835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:32835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:06.948000 3291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:06.949049 3291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:06.957517 3299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:06.957643 3291 server_base.cc:1034] running on GCE node
W20260927 18:47:06.957500 3296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:06.957563 3297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:06.958317 3291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:06.959313 3291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:06.960471 3291 hybrid_clock.cc:648] HybridClock initialized: now 1790534826960459 us; error 29 us; skew 500 ppm
I20260927 18:47:06.968658 3291 webserver.cc:460] Webserver started at http://127.29.159.126:37613/ using document root <none> and password file <none>
I20260927 18:47:06.969259 3291 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:06.969322 3291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:06.969559 3291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:06.971702 3291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "a6e30354ce2c495bac3107424a4ef87d"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "5edb8171dfa475d527c27c22f9ccb62b"
server_key_iv: "912977f0ddef7da1d721ae47f5e1b8af"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.972260 3291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "a6e30354ce2c495bac3107424a4ef87d"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "5edb8171dfa475d527c27c22f9ccb62b"
server_key_iv: "912977f0ddef7da1d721ae47f5e1b8af"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.975855 3291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:47:06.978209 3305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:06.979364 3291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:06.979538 3291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "a6e30354ce2c495bac3107424a4ef87d"
format_stamp: "Formatted at 2026-09-27 18:47:06 on dist-test-slave-8tg2"
server_key: "5edb8171dfa475d527c27c22f9ccb62b"
server_key_iv: "912977f0ddef7da1d721ae47f5e1b8af"
server_key_version: "encryptionkey@0"
I20260927 18:47:06.979642 3291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:07.001744 3291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:07.002473 3291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:07.002673 3291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:07.009974 3291 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:32835
I20260927 18:47:07.009984 3357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:32835 every 8 connection(s)
I20260927 18:47:07.010957 3291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:07.011655 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 3291
I20260927 18:47:07.011790 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:07.012100 30333 external_mini_cluster.cc:1428] Setting key 74f1ab5bf58e5fff0de85608d3e69c01
I20260927 18:47:07.014469 3358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:07.020119 3358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d: Bootstrap starting.
I20260927 18:47:07.021466 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:07.013491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.29.159.126:32835)
0927 18:47:07.014149 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:07.014164 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:07.014202 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:47:07.014893 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:07.014910 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:07.015307 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:07.015694 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:07.016113 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:07.017121 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:07.017994 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:07.018608 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:07.018773 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:07.019726 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:07.019731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:07.019743 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:07.020178 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:07.020279 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:07.020301 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:07.020404 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:07.020530 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 18:47:07.020781 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":146,"threads_started":1}
I20260927 18:47:07.022392 3358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:07.023236 3358 log.cc:824] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:07.024330 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:47:07.025354 3358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d: No bootstrap required, opened a new log
I20260927 18:47:07.028717 3358 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e30354ce2c495bac3107424a4ef87d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32835 } }
I20260927 18:47:07.028913 3358 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:07.028946 3358 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e30354ce2c495bac3107424a4ef87d, State: Initialized, Role: FOLLOWER
I20260927 18:47:07.029284 3358 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e30354ce2c495bac3107424a4ef87d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32835 } }
I20260927 18:47:07.029385 3358 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:07.029428 3358 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:07.029493 3358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:07.030333 3358 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e30354ce2c495bac3107424a4ef87d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32835 } }
I20260927 18:47:07.030609 3358 leader_election.cc:302] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6e30354ce2c495bac3107424a4ef87d; no voters:
I20260927 18:47:07.030843 3358 leader_election.cc:288] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:07.030982 3363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:07.031227 3363 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [term 1 LEADER]: Becoming Leader. State: Replica: a6e30354ce2c495bac3107424a4ef87d, State: Running, Role: LEADER
I20260927 18:47:07.031561 3363 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e30354ce2c495bac3107424a4ef87d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32835 } }
I20260927 18:47:07.032068 3358 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:07.033339 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6e30354ce2c495bac3107424a4ef87d. Latest consensus state: current_term: 1 leader_uuid: "a6e30354ce2c495bac3107424a4ef87d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e30354ce2c495bac3107424a4ef87d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32835 } } }
I20260927 18:47:07.033371 3364 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6e30354ce2c495bac3107424a4ef87d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e30354ce2c495bac3107424a4ef87d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32835 } } }
I20260927 18:47:07.033464 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:07.033468 3364 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:07.033973 3373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:07.036541 3373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:07.041491 3373 catalog_manager.cc:1347] Generated new cluster ID: 661b8c33326e43baa0f4fab36586d8b6
I20260927 18:47:07.041559 3373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:07.084676 3373 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:07.085630 3373 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:07.096277 3373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6e30354ce2c495bac3107424a4ef87d: Generated new TSK 0
I20260927 18:47:07.097034 3373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:47:07.199693 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:07.108639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.29.159.126:32835)
0927 18:47:07.108771 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:07.108775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:07.108921 (+ 146us) server_negotiation.cc:407] Connection header received
0927 18:47:07.109091 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:07.109093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:07.109139 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:07.109220 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:47:07.109478 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:07.109966 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:07.111225 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:07.111496 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:07.111574 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:07.111908 (+ 334us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:47:07.199245 (+ 87337us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:47:07.199574 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39114: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 18:47:07.199795 3361 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39114: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 18:47:07.199957 3387 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:07.108507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:32835 (local address 127.0.0.1:39114)
0927 18:47:07.108826 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:07.108841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:07.108955 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:07.109250 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:07.109253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:07.109261 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:47:07.109375 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:07.109382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:07.110100 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:07.110104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:07.111109 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:07.111119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:07.111618 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:07.111622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:07.111713 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:07.111741 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:47:07.199547 (+ 87806us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:47:07.199755 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.159.126:32835: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260927 18:47:07.200699 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 3291
2026-09-27T18:47:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (402 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (119 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:47:07 dist-test-slave-8tg2 krb5kdc[3399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:47:07 dist-test-slave-8tg2 krb5kdc[3399](info): set up 2 sockets
Sep 27 18:47:07 dist-test-slave-8tg2 krb5kdc[3399](info): commencing operation
krb5kdc: starting...
W20260927 18:47:08.471715 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:47:08 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:47:08.590333 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:35043
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:35043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:08.699604 3412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:08.699872 3412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:08.699911 3412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:08.703780 3412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:08.703832 3412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:08.703851 3412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:08.703867 3412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:08.703882 3412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:08.709712 3412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:35043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:35043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:08.710687 3412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:08.711723 3412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:08.719957 3418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:08.719970 3417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:08.720089 3420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:08.720299 3412 server_base.cc:1034] running on GCE node
I20260927 18:47:08.721107 3412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:08.722486 3412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:08.723682 3412 hybrid_clock.cc:648] HybridClock initialized: now 1790534828723663 us; error 34 us; skew 500 ppm
Sep 27 18:47:08 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534828, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:08.726902 3412 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:47:08.727771 3412 webserver.cc:460] Webserver started at http://127.29.159.126:36349/ using document root <none> and password file <none>
I20260927 18:47:08.728273 3412 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:08.728329 3412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:08.728487 3412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:08.730432 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "09319d42484b4bb5982b27b840c6d843"
format_stamp: "Formatted at 2026-09-27 18:47:08 on dist-test-slave-8tg2"
server_key: "d0169ded394f07ef42a0dc950a74f1b0"
server_key_iv: "0872453070c8092296ee1bbefea7076f"
server_key_version: "encryptionkey@0"
I20260927 18:47:08.730918 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "09319d42484b4bb5982b27b840c6d843"
format_stamp: "Formatted at 2026-09-27 18:47:08 on dist-test-slave-8tg2"
server_key: "d0169ded394f07ef42a0dc950a74f1b0"
server_key_iv: "0872453070c8092296ee1bbefea7076f"
server_key_version: "encryptionkey@0"
I20260927 18:47:08.734419 3412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 18:47:08.736974 3427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:08.738164 3412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:47:08.738329 3412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "09319d42484b4bb5982b27b840c6d843"
format_stamp: "Formatted at 2026-09-27 18:47:08 on dist-test-slave-8tg2"
server_key: "d0169ded394f07ef42a0dc950a74f1b0"
server_key_iv: "0872453070c8092296ee1bbefea7076f"
server_key_version: "encryptionkey@0"
I20260927 18:47:08.738437 3412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:08.758711 3412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:08.761729 3412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:08.761942 3412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:08.768756 3412 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:35043
I20260927 18:47:08.768853 3479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:35043 every 8 connection(s)
I20260927 18:47:08.769740 3412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:08.772491 3480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:08.776356 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 3412
I20260927 18:47:08.776489 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:08.776783 30333 external_mini_cluster.cc:1428] Setting key fa3cb7c713652dc5688af6bf205edb9a
I20260927 18:47:08.778040 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843: Bootstrap starting.
I20260927 18:47:08.780719 3480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:08.781589 3480 log.cc:824] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:08.783562 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843: No bootstrap required, opened a new log
Sep 27 18:47:08 dist-test-slave-8tg2 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:08.787341 3480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09319d42484b4bb5982b27b840c6d843" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 35043 } }
I20260927 18:47:08.787627 3480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:08.787691 3480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09319d42484b4bb5982b27b840c6d843, State: Initialized, Role: FOLLOWER
I20260927 18:47:08.788156 3480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09319d42484b4bb5982b27b840c6d843" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 35043 } }
I20260927 18:47:08.788295 3480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:08.788383 3480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:08.788484 3480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:08.789369 3480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09319d42484b4bb5982b27b840c6d843" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 35043 } }
I20260927 18:47:08.789734 3480 leader_election.cc:302] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09319d42484b4bb5982b27b840c6d843; no voters:
I20260927 18:47:08.790091 3480 leader_election.cc:288] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:08.790194 3485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:08.790395 3485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [term 1 LEADER]: Becoming Leader. State: Replica: 09319d42484b4bb5982b27b840c6d843, State: Running, Role: LEADER
I20260927 18:47:08.790735 3485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09319d42484b4bb5982b27b840c6d843" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 35043 } }
I20260927 18:47:08.791410 3480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:08.791988 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09319d42484b4bb5982b27b840c6d843. Latest consensus state: current_term: 1 leader_uuid: "09319d42484b4bb5982b27b840c6d843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09319d42484b4bb5982b27b840c6d843" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 35043 } } }
I20260927 18:47:08.792102 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:08.792652 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:08.778246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.29.159.126:35043)
0927 18:47:08.778635 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:08.778654 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:08.778694 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:47:08.779569 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:08.779592 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:08.780006 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:08.780366 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:08.781098 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:08.782419 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:08.783306 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:08.783991 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:08.784146 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:08.786594 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:08.786601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:08.786618 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:08.786647 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:08.788819 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:08.789278 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:08.789284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:08.789290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:08.789409 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:08.789776 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:08.789779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:08.789780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:08.790362 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:08.790569 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:08.791137 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 18:47:08.791465 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
I20260927 18:47:08.792734 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09319d42484b4bb5982b27b840c6d843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09319d42484b4bb5982b27b840c6d843" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 35043 } } }
I20260927 18:47:08.792975 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:08.793130 3491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:08.796799 3491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:08.802173 3491 catalog_manager.cc:1347] Generated new cluster ID: 84085262f96148f2b3a2dc802e59f6b7
I20260927 18:47:08.802242 3491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:08.823872 3491 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:08.824990 3491 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:08.828956 3491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09319d42484b4bb5982b27b840c6d843: Generated new TSK 0
I20260927 18:47:08.829622 3491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:47:08.866580 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:35043
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:08.975896 3506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:08.976167 3506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:08.976210 3506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:08.980261 3506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:08.980319 3506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:08.980409 3506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:08.985594 3506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:35043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:08.986618 3506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:08.987656 3506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:47:08.997442 3506 server_base.cc:1034] running on GCE node
W20260927 18:47:08.997467 3511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:08.997404 3512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:08.997622 3514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:08.998027 3506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:08.998685 3506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:08.999843 3506 hybrid_clock.cc:648] HybridClock initialized: now 1790534828999829 us; error 30 us; skew 500 ppm
Sep 27 18:47:08 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534828, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:09.002584 3506 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:47:09.003549 3506 webserver.cc:460] Webserver started at http://127.29.159.65:43929/ using document root <none> and password file <none>
I20260927 18:47:09.004302 3506 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:09.004379 3506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:09.004621 3506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:09.006805 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "7e2286becd6e4e89a1cb06fc70125676"
format_stamp: "Formatted at 2026-09-27 18:47:09 on dist-test-slave-8tg2"
server_key: "f692d888e2c4c38fb431e8a28f00e11e"
server_key_iv: "6330ac287cf1afb870aeab04984f6bdf"
server_key_version: "encryptionkey@0"
I20260927 18:47:09.007396 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "7e2286becd6e4e89a1cb06fc70125676"
format_stamp: "Formatted at 2026-09-27 18:47:09 on dist-test-slave-8tg2"
server_key: "f692d888e2c4c38fb431e8a28f00e11e"
server_key_iv: "6330ac287cf1afb870aeab04984f6bdf"
server_key_version: "encryptionkey@0"
I20260927 18:47:09.011197 3506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:47:09.013798 3521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:09.015036 3506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:09.015131 3506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "7e2286becd6e4e89a1cb06fc70125676"
format_stamp: "Formatted at 2026-09-27 18:47:09 on dist-test-slave-8tg2"
server_key: "f692d888e2c4c38fb431e8a28f00e11e"
server_key_iv: "6330ac287cf1afb870aeab04984f6bdf"
server_key_version: "encryptionkey@0"
I20260927 18:47:09.015228 3506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:09.029584 3506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:09.032743 3506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:09.032895 3506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:09.033425 3506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:09.034214 3506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:09.034260 3506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:09.034301 3506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:09.034317 3506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:09.043606 3634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:38947 every 8 connection(s)
I20260927 18:47:09.043587 3506 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:38947
I20260927 18:47:09.044661 3506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:09.052960 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 3506
I20260927 18:47:09.053097 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:47:09.053404 30333 external_mini_cluster.cc:1428] Setting key dcb8f2a2c8eee9a59e1bc288a52acb34
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534828, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:09.059849 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:09.046763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:48547 (local address 127.29.159.126:35043)
0927 18:47:09.046963 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:09.046967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:09.047950 (+ 983us) server_negotiation.cc:407] Connection header received
0927 18:47:09.049311 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:09.049314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:09.049365 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:09.049450 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:09.050692 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.051286 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.052561 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.052833 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.052928 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:09.055963 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:09.055967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:09.055970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:09.055997 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:09.057532 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.058046 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.058050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.058052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.058197 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.058671 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.058674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.058676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.058871 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:09.058988 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:09.059528 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 18:47:09.059657 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 18:47:09.060730 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:09.047238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:35043 (local address 127.29.159.65:48547)
0927 18:47:09.047787 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:09.047833 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:09.049087 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:09.049658 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:09.049665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:09.050035 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:09.050533 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:09.050546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.051493 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.051496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:09.052437 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:09.052446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.053016 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.053023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:09.053256 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:09.053942 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:09.053964 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:09.055800 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:09.057657 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:09.057662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:09.057678 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:09.057938 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:09.058392 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:09.058398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:09.058401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:09.058563 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:09.058987 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:09.058995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:09.059255 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 18:47:09.059461 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 18:47:09.059806 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":170,"threads_started":1}
I20260927 18:47:09.061982 3635 heartbeater.cc:347] Connected to a master server at 127.29.159.126:35043
I20260927 18:47:09.062275 3635 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:09.062919 3635 heartbeater.cc:511] Master 127.29.159.126:35043 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:47:09.064603 3444 ts_manager.cc:194] Registered new tserver with Master: 7e2286becd6e4e89a1cb06fc70125676 (127.29.159.65:38947)
I20260927 18:47:09.065814 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:48547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:47:09.081981 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:35043
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:09.196568 3640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:09.196820 3640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:09.196864 3640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:09.201030 3640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:09.201102 3640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:09.201162 3640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:09.206246 3640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:35043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:09.207261 3640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:09.208361 3640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:09.218055 3646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:09.218051 3648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:09.218127 3640 server_base.cc:1034] running on GCE node
W20260927 18:47:09.218367 3645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:09.218801 3640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:09.219523 3640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:09.220685 3640 hybrid_clock.cc:648] HybridClock initialized: now 1790534829220666 us; error 32 us; skew 500 ppm
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:09.223353 3640 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:47:09.224400 3640 webserver.cc:460] Webserver started at http://127.29.159.66:45719/ using document root <none> and password file <none>
I20260927 18:47:09.224969 3640 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:09.225030 3640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:09.225274 3640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:09.227300 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "c50d65b034c14d28bb95994a48c39142"
format_stamp: "Formatted at 2026-09-27 18:47:09 on dist-test-slave-8tg2"
server_key: "59d33a444d01552c5960439755e4d30b"
server_key_iv: "58a7d8dfb993f82c2c6ad4652ff5bcf1"
server_key_version: "encryptionkey@0"
I20260927 18:47:09.227944 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "c50d65b034c14d28bb95994a48c39142"
format_stamp: "Formatted at 2026-09-27 18:47:09 on dist-test-slave-8tg2"
server_key: "59d33a444d01552c5960439755e4d30b"
server_key_iv: "58a7d8dfb993f82c2c6ad4652ff5bcf1"
server_key_version: "encryptionkey@0"
I20260927 18:47:09.231709 3640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:47:09.234261 3655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:09.235560 3640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:09.235709 3640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "c50d65b034c14d28bb95994a48c39142"
format_stamp: "Formatted at 2026-09-27 18:47:09 on dist-test-slave-8tg2"
server_key: "59d33a444d01552c5960439755e4d30b"
server_key_iv: "58a7d8dfb993f82c2c6ad4652ff5bcf1"
server_key_version: "encryptionkey@0"
I20260927 18:47:09.235822 3640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:09.265931 3640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:09.269079 3640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:09.269297 3640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:09.269914 3640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:09.270747 3640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:09.270819 3640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:09.270890 3640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:09.270943 3640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:09.279909 3640 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:45811
I20260927 18:47:09.279939 3768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:45811 every 8 connection(s)
I20260927 18:47:09.280812 3640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:09.288833 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 3640
I20260927 18:47:09.288987 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:09.289335 30333 external_mini_cluster.cc:1428] Setting key 73f9106e672b7f06734a69bd7fcef921
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:09.295337 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:09.282680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:47691 (local address 127.29.159.126:35043)
0927 18:47:09.282818 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:09.282827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:09.283650 (+ 823us) server_negotiation.cc:407] Connection header received
0927 18:47:09.285064 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:09.285067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:09.285127 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:09.285208 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:09.286374 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.286914 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.288076 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.288314 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.288428 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:09.291164 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:09.291167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:09.291169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:09.291192 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:09.292999 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.293644 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.293646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.293648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.293722 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.294129 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.294135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.294139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.294374 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:09.294477 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:09.294958 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 18:47:09.295096 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:47:09.296101 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:09.283011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:35043 (local address 127.29.159.66:47691)
0927 18:47:09.283481 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:09.283524 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:09.284843 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:09.285382 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:09.285389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:09.285743 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:09.286222 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:09.286235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.287044 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.287048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:09.287950 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:09.287959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.288464 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.288469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:09.288642 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:09.289295 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:09.289311 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:09.291002 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:09.293133 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:09.293141 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:09.293160 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:09.293534 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:09.293826 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:09.293831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:09.293834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:09.294020 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:09.294476 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:09.294481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:09.294708 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 18:47:09.294879 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 18:47:09.295130 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":98,"threads_started":1}
I20260927 18:47:09.297320 3769 heartbeater.cc:347] Connected to a master server at 127.29.159.126:35043
I20260927 18:47:09.297573 3769 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:09.298061 3769 heartbeater.cc:511] Master 127.29.159.126:35043 requested a full tablet report, sending...
I20260927 18:47:09.299083 3444 ts_manager.cc:194] Registered new tserver with Master: c50d65b034c14d28bb95994a48c39142 (127.29.159.66:45811)
I20260927 18:47:09.299707 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:47691
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:47:09.316552 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:35043
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:09.427907 3774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:09.428148 3774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:09.428189 3774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:09.432284 3774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:09.432332 3774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:09.432416 3774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:09.437486 3774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:35043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:09.438503 3774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:09.439604 3774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:09.449105 3782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:09.449111 3779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:09.449115 3780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:09.449431 3774 server_base.cc:1034] running on GCE node
I20260927 18:47:09.449892 3774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:09.450497 3774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:09.451649 3774 hybrid_clock.cc:648] HybridClock initialized: now 1790534829451634 us; error 32 us; skew 500 ppm
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:09.454334 3774 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:47:09.455206 3774 webserver.cc:460] Webserver started at http://127.29.159.67:44427/ using document root <none> and password file <none>
I20260927 18:47:09.455824 3774 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:09.455888 3774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:09.456135 3774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:09.458118 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "ac8cd1a827154c80bfda3ba8b285416e"
format_stamp: "Formatted at 2026-09-27 18:47:09 on dist-test-slave-8tg2"
server_key: "be04c56de9ad207a4850fe23b1e2e42c"
server_key_iv: "35f4bc72c76966c18473e7419fa5d71f"
server_key_version: "encryptionkey@0"
I20260927 18:47:09.458670 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "ac8cd1a827154c80bfda3ba8b285416e"
format_stamp: "Formatted at 2026-09-27 18:47:09 on dist-test-slave-8tg2"
server_key: "be04c56de9ad207a4850fe23b1e2e42c"
server_key_iv: "35f4bc72c76966c18473e7419fa5d71f"
server_key_version: "encryptionkey@0"
I20260927 18:47:09.462293 3774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 18:47:09.464685 3789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:09.465909 3774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:47:09.466063 3774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "ac8cd1a827154c80bfda3ba8b285416e"
format_stamp: "Formatted at 2026-09-27 18:47:09 on dist-test-slave-8tg2"
server_key: "be04c56de9ad207a4850fe23b1e2e42c"
server_key_iv: "35f4bc72c76966c18473e7419fa5d71f"
server_key_version: "encryptionkey@0"
I20260927 18:47:09.466197 3774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:09.479107 3774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:09.481989 3774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:09.482194 3774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:09.482810 3774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:09.483650 3774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:09.483721 3774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:09.483789 3774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:09.483837 3774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:09.492844 3774 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:37317
I20260927 18:47:09.492945 3902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:37317 every 8 connection(s)
I20260927 18:47:09.493775 3774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:09.503278 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 3774
I20260927 18:47:09.503398 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:09.503712 30333 external_mini_cluster.cc:1428] Setting key 942eef47c3870a50627ad4099bc8ce06
I20260927 18:47:09.507587 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:09.495550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:39813 (local address 127.29.159.126:35043)
0927 18:47:09.495702 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:09.495705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:09.496585 (+ 880us) server_negotiation.cc:407] Connection header received
0927 18:47:09.498075 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:09.498079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:09.498150 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:09.498257 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:09.499410 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.499973 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.501101 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.501327 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.501438 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:09.503970 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:09.503974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:09.503976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:09.503999 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:09.505581 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.506098 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.506103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.506104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.506192 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.506529 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.506534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.506536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.506714 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:09.506821 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:09.507292 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 18:47:09.507403 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:47:09.508328 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:09.495975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:35043 (local address 127.29.159.67:39813)
0927 18:47:09.496423 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:09.496464 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:09.497857 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:09.498446 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:09.498452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:09.498799 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:09.499263 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:09.499277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.500104 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.500107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:09.500975 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:09.500984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.501477 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.501481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:09.501650 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:09.502221 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:09.502236 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:09.503790 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:09.505722 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:09.505728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:09.505741 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:09.505991 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:09.506292 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:09.506295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:09.506297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:09.506435 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:09.506819 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:09.506825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:09.507058 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 18:47:09.507237 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 18:47:09.507566 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":100,"threads_started":1}
I20260927 18:47:09.509425 3903 heartbeater.cc:347] Connected to a master server at 127.29.159.126:35043
I20260927 18:47:09.509672 3903 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:09.510129 3903 heartbeater.cc:511] Master 127.29.159.126:35043 requested a full tablet report, sending...
I20260927 18:47:09.511081 3444 ts_manager.cc:194] Registered new tserver with Master: ac8cd1a827154c80bfda3ba8b285416e (127.29.159.67:37317)
I20260927 18:47:09.511729 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:39813
I20260927 18:47:09.517525 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/test-admin.keytab.
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:09.533320 30333 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 18:47:09.536399 3910 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:09.534860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.29.159.65:38947)
0927 18:47:09.535192 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:09.535209 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:09.535239 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:47:09.535310 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:09.535315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:09.535569 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:09.535728 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:09.536202 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46578: BlockingRecv error: recv got EOF from 127.0.0.1:46578 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":63,"threads_started":1}
W20260927 18:47:09.536633 30333 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5cc)
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:09.537705 30333 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:09.538728 30333 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:47:09.548465 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:09.539710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46586 (local address 127.29.159.65:38947)
0927 18:47:09.539855 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:09.539860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:09.540009 (+ 149us) server_negotiation.cc:407] Connection header received
0927 18:47:09.540101 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:09.540103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:09.540143 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:09.540209 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:09.540728 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.541686 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.542535 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.542935 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.543024 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:09.545226 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:09.545234 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:09.545247 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:09.545269 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:09.546799 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.547192 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.547198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.547199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.547276 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.547540 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.547542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.547544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.547805 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:09.547973 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:09.548119 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 18:47:09.548304 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 18:47:09.552291 3910 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:09.551293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.29.159.65:38947)
0927 18:47:09.551465 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:09.551470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:09.551508 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:47:09.551615 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:09.551617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:09.551650 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:09.551716 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:09.552181 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46602: BlockingRecv error: recv got EOF from 127.0.0.1:46602 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 18:47:09.552350 30333 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:09.553335 30333 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:09.554206 30333 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:47:09.562569 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:09.554982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.29.159.65:38947)
0927 18:47:09.555108 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:09.555111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:09.555175 (+ 64us) server_negotiation.cc:407] Connection header received
0927 18:47:09.555264 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:09.555266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:09.555300 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:09.555363 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:09.555898 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.556400 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.557222 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.557444 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.557517 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:09.559728 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:09.559732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:09.559734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:09.559755 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:09.561136 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.561511 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.561513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.561515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.561583 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.561803 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.561805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.561806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.561949 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:09.562042 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:09.562285 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 18:47:09.562406 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 18:47:09.565248 3910 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:09.564227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.29.159.65:38947)
0927 18:47:09.564346 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:09.564350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:09.564441 (+ 91us) server_negotiation.cc:407] Connection header received
0927 18:47:09.564576 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:09.564579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:09.564626 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:09.564707 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:09.565077 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46620: BlockingRecv error: recv got EOF from 127.0.0.1:46620 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 18:47:09.565330 30333 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534801142164-30333-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:09.566339 30333 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:09.567209 30333 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:47:09.575255 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:09.567990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.29.159.65:38947)
0927 18:47:09.568096 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:09.568100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:09.568240 (+ 140us) server_negotiation.cc:407] Connection header received
0927 18:47:09.568306 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:09.568308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:09.568344 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:09.568408 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:09.568845 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.569355 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.570143 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:09.570343 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:09.570538 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:09.572628 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:09.572631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:09.572634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:09.572653 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:09.573855 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.574222 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.574225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.574226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.574295 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:09.574513 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:09.574515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:09.574516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:09.574655 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:09.574773 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:09.574954 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 18:47:09.575079 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:47:09.576229 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 3506
I20260927 18:47:09.581753 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 3640
I20260927 18:47:09.587090 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 3774
I20260927 18:47:09.592299 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 3412
2026-09-27T18:47:09Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2154 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3923](info): set up 2 sockets
Sep 27 18:47:09 dist-test-slave-8tg2 krb5kdc[3923](info): commencing operation
krb5kdc: starting...
W20260927 18:47:10.623883 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:47:10 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:47:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.126.keytab.
Entry for principal oryx/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.126.keytab.
I20260927 18:47:10.762193 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:40221
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.126.keytab
--principal=oryx/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:40221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:10.869346 3936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:10.869592 3936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:10.869659 3936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:10.873616 3936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:10.873667 3936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:10.873687 3936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:10.873744 3936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:10.873764 3936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:10.878841 3936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:40221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.126.keytab
--principal=oryx/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:40221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:10.879895 3936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:10.880864 3936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:10.889362 3941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:10.889362 3942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:10.889495 3936 server_base.cc:1034] running on GCE node
W20260927 18:47:10.889461 3944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:10.890169 3936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:10.891271 3936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:10.892459 3936 hybrid_clock.cc:648] HybridClock initialized: now 1790534830892441 us; error 41 us; skew 500 ppm
Sep 27 18:47:10 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534830, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:10.895133 3936 init.cc:377] Logged in from keytab as oryx/127.29.159.126@KRBTEST.COM (short username oryx)
I20260927 18:47:10.896073 3936 webserver.cc:460] Webserver started at http://127.29.159.126:41845/ using document root <none> and password file <none>
I20260927 18:47:10.896620 3936 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:10.896721 3936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:10.896976 3936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:10.899066 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "1af467bdd4fc4d3f90cf79667730d4f3"
format_stamp: "Formatted at 2026-09-27 18:47:10 on dist-test-slave-8tg2"
server_key: "20193e4c8ed4399d4d2527cede95c606"
server_key_iv: "c437bab5efdf328cb087c9afa8486703"
server_key_version: "encryptionkey@0"
I20260927 18:47:10.899658 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "1af467bdd4fc4d3f90cf79667730d4f3"
format_stamp: "Formatted at 2026-09-27 18:47:10 on dist-test-slave-8tg2"
server_key: "20193e4c8ed4399d4d2527cede95c606"
server_key_iv: "c437bab5efdf328cb087c9afa8486703"
server_key_version: "encryptionkey@0"
I20260927 18:47:10.903169 3936 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:47:10.905735 3951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:10.906981 3936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:10.907131 3936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "1af467bdd4fc4d3f90cf79667730d4f3"
format_stamp: "Formatted at 2026-09-27 18:47:10 on dist-test-slave-8tg2"
server_key: "20193e4c8ed4399d4d2527cede95c606"
server_key_iv: "c437bab5efdf328cb087c9afa8486703"
server_key_version: "encryptionkey@0"
I20260927 18:47:10.907258 3936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:10.927531 3936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:10.930716 3936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:10.930923 3936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:10.937902 3936 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:40221
I20260927 18:47:10.938179 4003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:40221 every 8 connection(s)
I20260927 18:47:10.938943 3936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:10.941725 4004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:10.946966 4004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3: Bootstrap starting.
I20260927 18:47:10.948115 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 3936
I20260927 18:47:10.948273 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:10.948593 30333 external_mini_cluster.cc:1428] Setting key 0a331466a4fe13b7670f0de4f4bfec2c
I20260927 18:47:10.949772 4004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:10.950517 4004 log.cc:824] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:10.952749 4004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3: No bootstrap required, opened a new log
I20260927 18:47:10.957057 4004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } }
I20260927 18:47:10.957429 4004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:10.957494 4004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af467bdd4fc4d3f90cf79667730d4f3, State: Initialized, Role: FOLLOWER
Sep 27 18:47:10 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:10.957952 4004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } }
I20260927 18:47:10.958142 4004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:10.958232 4004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:10.958336 4004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:10.959364 4004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } }
I20260927 18:47:10.959724 4004 leader_election.cc:302] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af467bdd4fc4d3f90cf79667730d4f3; no voters:
I20260927 18:47:10.960058 4004 leader_election.cc:288] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:10.960122 4009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:10.960335 4009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 1 LEADER]: Becoming Leader. State: Replica: 1af467bdd4fc4d3f90cf79667730d4f3, State: Running, Role: LEADER
I20260927 18:47:10.960614 4009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } }
I20260927 18:47:10.961314 4004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:10.962301 4010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af467bdd4fc4d3f90cf79667730d4f3. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } }
I20260927 18:47:10.962704 4010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:10.962610 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } }
I20260927 18:47:10.963493 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:10.964643 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:10.949778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.29.159.126:40221)
0927 18:47:10.950240 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:10.950267 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:10.950305 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:47:10.951075 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:10.951099 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:10.951546 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:10.951830 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:10.952714 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:10.953734 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:10.954898 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:10.955372 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:10.955558 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:10.958661 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:10.958666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:10.958679 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:10.958705 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:10.960822 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:10.961301 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:10.961307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:10.961313 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:10.961477 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:10.961782 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:10.961786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:10.961787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:10.962340 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:10.962524 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:10.963450 (+ 926us) server_negotiation.cc:299] Negotiation successful
0927 18:47:10.963759 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":153,"threads_started":1}
W20260927 18:47:10.966692 4025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:47:10.966770 4025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:47:10.967061 4020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:10.967943 4020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:10.973310 4020 catalog_manager.cc:1347] Generated new cluster ID: 3cd05ce26e8f4e80ac24854feb0b86a2
I20260927 18:47:10.973362 4020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:11.001279 4020 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:11.002279 4020 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:11.010697 4020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3: Generated new TSK 0
I20260927 18:47:11.011472 4020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab.
Entry for principal oryx/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab.
I20260927 18:47:11.040805 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:40221
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab
--principal=oryx/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:11.157332 4030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:11.157613 4030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:11.157693 4030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:11.161973 4030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:11.162029 4030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:11.162129 4030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:11.167755 4030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab
--principal=oryx/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:40221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:11.168862 4030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:11.169902 4030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:11.179335 4038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:11.179409 4036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:11.179617 4030 server_base.cc:1034] running on GCE node
W20260927 18:47:11.179404 4035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:11.180030 4030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:11.180612 4030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:11.181770 4030 hybrid_clock.cc:648] HybridClock initialized: now 1790534831181756 us; error 27 us; skew 500 ppm
Sep 27 18:47:11 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534831, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:11.184600 4030 init.cc:377] Logged in from keytab as oryx/127.29.159.65@KRBTEST.COM (short username oryx)
I20260927 18:47:11.185465 4030 webserver.cc:460] Webserver started at http://127.29.159.65:35513/ using document root <none> and password file <none>
I20260927 18:47:11.186028 4030 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:11.186088 4030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:11.186299 4030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:11.188378 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "cf85e20a1ec8466ca9bcd09b7df90878"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "8e36ef3de2464270b77f4e9404109864"
server_key_iv: "fa47468b341595f9533225219b4c9a0a"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.188927 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "cf85e20a1ec8466ca9bcd09b7df90878"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "8e36ef3de2464270b77f4e9404109864"
server_key_iv: "fa47468b341595f9533225219b4c9a0a"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.192445 4030 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:47:11.194707 4045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:11.195955 4030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:11.196089 4030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "cf85e20a1ec8466ca9bcd09b7df90878"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "8e36ef3de2464270b77f4e9404109864"
server_key_iv: "fa47468b341595f9533225219b4c9a0a"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.196185 4030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:11.208132 4030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:11.210996 4030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:11.211185 4030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:11.211822 4030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:11.212636 4030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:11.212682 4030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:11.212719 4030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:11.212741 4030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:11.221252 4030 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:45531
I20260927 18:47:11.221280 4158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:45531 every 8 connection(s)
I20260927 18:47:11.222115 4030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:11.227209 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 4030
I20260927 18:47:11.227358 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:47:11.227677 30333 external_mini_cluster.cc:1428] Setting key a41cc517c86c685a9d5564be2e3ab24e
Sep 27 18:47:11 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534831, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:11.237356 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.223946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:54935 (local address 127.29.159.126:40221)
0927 18:47:11.224072 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:11.224076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:11.224855 (+ 779us) server_negotiation.cc:407] Connection header received
0927 18:47:11.226237 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:11.226241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:11.226303 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:11.226408 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:11.227698 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.228469 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.229876 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.230130 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.230246 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.233048 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:11.233052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:11.233057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:11.233087 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:11.234887 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.235514 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.235520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.235521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.235623 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.236007 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.236010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.236012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.236280 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:11.236394 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:11.236954 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 18:47:11.237113 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for oryx/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:47:11.238032 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.224244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.65:54935)
0927 18:47:11.224681 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:11.224722 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:11.226017 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:11.226584 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:11.226591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:11.226944 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:11.227541 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.227566 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.228601 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.228605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.229748 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.229758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.230284 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.230289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.230483 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.231115 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:11.231132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:11.232875 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:11.235014 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.235019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.235033 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.235356 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.235734 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.235737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.235738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.235891 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.236396 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:11.236401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:11.236650 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 18:47:11.236825 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 18:47:11.237102 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260927 18:47:11.239360 4159 heartbeater.cc:347] Connected to a master server at 127.29.159.126:40221
I20260927 18:47:11.239678 4159 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.66@KRBTEST.COM" created.
I20260927 18:47:11.240252 4159 heartbeater.cc:511] Master 127.29.159.126:40221 requested a full tablet report, sending...
I20260927 18:47:11.241832 3968 ts_manager.cc:194] Registered new tserver with Master: cf85e20a1ec8466ca9bcd09b7df90878 (127.29.159.65:45531)
I20260927 18:47:11.242936 3968 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:54935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab.
Entry for principal oryx/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab.
I20260927 18:47:11.258916 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:40221
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab
--principal=oryx/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:11.373526 4164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:11.373817 4164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:11.373894 4164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:11.378346 4164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:11.378454 4164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:11.378544 4164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:11.384066 4164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab
--principal=oryx/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:40221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:11.385073 4164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:11.386063 4164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:47:11.396935 4164 server_base.cc:1034] running on GCE node
W20260927 18:47:11.396891 4169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:11.396911 4170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:11.397123 4172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:11.397519 4164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:11.398181 4164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:11.399375 4164 hybrid_clock.cc:648] HybridClock initialized: now 1790534831399360 us; error 29 us; skew 500 ppm
Sep 27 18:47:11 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534831, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:11.402064 4164 init.cc:377] Logged in from keytab as oryx/127.29.159.66@KRBTEST.COM (short username oryx)
I20260927 18:47:11.402918 4164 webserver.cc:460] Webserver started at http://127.29.159.66:46295/ using document root <none> and password file <none>
I20260927 18:47:11.403527 4164 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:11.403618 4164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:11.403856 4164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:11.405848 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "d4346c5843c646e4bbf1b621a5d0c875"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "77ae633b01834423c54c945afa5d5d6e"
server_key_iv: "9a566938b3ba05b9fbec16d9ed03f3db"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.406410 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "d4346c5843c646e4bbf1b621a5d0c875"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "77ae633b01834423c54c945afa5d5d6e"
server_key_iv: "9a566938b3ba05b9fbec16d9ed03f3db"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.409946 4164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 18:47:11.412312 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:11.413412 4164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:11.413542 4164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "d4346c5843c646e4bbf1b621a5d0c875"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "77ae633b01834423c54c945afa5d5d6e"
server_key_iv: "9a566938b3ba05b9fbec16d9ed03f3db"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.413659 4164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:11.449477 4164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:11.452453 4164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:11.452601 4164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:11.453146 4164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:11.453974 4164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:11.454038 4164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:11.454126 4164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:11.454165 4164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:11.463353 4164 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:43979
I20260927 18:47:11.463395 4292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:43979 every 8 connection(s)
I20260927 18:47:11.464355 4164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:11.469101 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 4164
I20260927 18:47:11.469192 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:11.469468 30333 external_mini_cluster.cc:1428] Setting key 5d8449112ba96e09ef66be70d0777744
Sep 27 18:47:11 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534831, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:47:11.478698 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.466121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:53221 (local address 127.29.159.126:40221)
0927 18:47:11.466273 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:11.466277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:11.467129 (+ 852us) server_negotiation.cc:407] Connection header received
0927 18:47:11.468589 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:11.468592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:11.468649 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:11.468725 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:11.470043 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.470725 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.471984 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.472368 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.472484 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.475131 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:11.475135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:11.475137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:11.475171 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:11.476665 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.477172 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.477177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.477178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.477266 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.477608 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.477612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.477614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.477801 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:11.477910 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:11.478413 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 18:47:11.478532 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:47:11.479581 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.466509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.66:53221)
0927 18:47:11.466955 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:11.467017 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:11.468366 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:11.468894 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:11.468901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:11.469313 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:11.469886 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.469905 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.470877 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.470880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.471865 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.471874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.472539 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.472544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.472787 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.473443 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:11.473466 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:11.474966 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:11.476788 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.476793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.476809 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.477065 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.477368 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.477372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.477374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.477511 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.477917 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:11.477926 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:11.478178 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 18:47:11.478373 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 18:47:11.478686 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260927 18:47:11.480783 4293 heartbeater.cc:347] Connected to a master server at 127.29.159.126:40221
I20260927 18:47:11.481037 4293 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.67@KRBTEST.COM" created.
I20260927 18:47:11.481628 4293 heartbeater.cc:511] Master 127.29.159.126:40221 requested a full tablet report, sending...
I20260927 18:47:11.482729 3968 ts_manager.cc:194] Registered new tserver with Master: d4346c5843c646e4bbf1b621a5d0c875 (127.29.159.66:43979)
I20260927 18:47:11.483294 3968 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:53221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab.
Entry for principal oryx/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab.
I20260927 18:47:11.498019 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:40221
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab
--principal=oryx/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:11.606935 4298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:11.607162 4298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:11.607203 4298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:11.611138 4298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:11.611189 4298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:11.611244 4298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:11.616281 4298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab
--principal=oryx/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:40221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:11.617197 4298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:11.618197 4298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:11.627498 4304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:11.627444 4303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:11.627605 4298 server_base.cc:1034] running on GCE node
W20260927 18:47:11.627429 4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:11.628119 4298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:11.628724 4298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:11.629913 4298 hybrid_clock.cc:648] HybridClock initialized: now 1790534831629882 us; error 24 us; skew 500 ppm
Sep 27 18:47:11 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534831, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:11.632644 4298 init.cc:377] Logged in from keytab as oryx/127.29.159.67@KRBTEST.COM (short username oryx)
I20260927 18:47:11.633394 4298 webserver.cc:460] Webserver started at http://127.29.159.67:38977/ using document root <none> and password file <none>
I20260927 18:47:11.633881 4298 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:11.633936 4298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:11.634091 4298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:11.636098 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "7fae20bb731f4ce2bea52eaa0418c7d1"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "630d5b9cd47d6f3d5b3ca061485c95df"
server_key_iv: "ab827eb36ef1224ab18ef420e1ab2251"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.636596 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "7fae20bb731f4ce2bea52eaa0418c7d1"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "630d5b9cd47d6f3d5b3ca061485c95df"
server_key_iv: "ab827eb36ef1224ab18ef420e1ab2251"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.640123 4298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:47:11.642379 4313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:11.643671 4298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:11.643798 4298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "7fae20bb731f4ce2bea52eaa0418c7d1"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "630d5b9cd47d6f3d5b3ca061485c95df"
server_key_iv: "ab827eb36ef1224ab18ef420e1ab2251"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.643908 4298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:11.655038 4298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:11.657850 4298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:11.658054 4298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:11.658649 4298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:11.659408 4298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:11.659534 4298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:11.659600 4298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:11.659655 4298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:11.668334 4298 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:35793
I20260927 18:47:11.668355 4426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:35793 every 8 connection(s)
I20260927 18:47:11.669154 4298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:11.674127 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 4298
I20260927 18:47:11.674255 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:11.674552 30333 external_mini_cluster.cc:1428] Setting key 492771b6fe57451771168a4b6276bff5
Sep 27 18:47:11 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534831, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:11.682739 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.670907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:47191 (local address 127.29.159.126:40221)
0927 18:47:11.671035 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:11.671039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:11.671836 (+ 797us) server_negotiation.cc:407] Connection header received
0927 18:47:11.673244 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:11.673248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:11.673303 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:11.673407 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:11.674803 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.675483 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.676906 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.677118 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.677197 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.679490 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:11.679493 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:11.679495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:11.679518 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:11.681011 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.681449 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.681452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.681453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.681534 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.681853 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.681856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.681857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.682030 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:11.682111 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:11.682533 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 18:47:11.682625 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:47:11.683534 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.671259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.67:47191)
0927 18:47:11.671698 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:11.671738 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:11.672998 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:11.673569 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:11.673575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:11.673969 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:11.674629 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.674650 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.675606 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.675610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.676792 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.676808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.677241 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.677244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.677375 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.677894 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:11.677909 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:11.679270 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:11.681116 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.681121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.681132 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.681362 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.681625 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.681627 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.681628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.681753 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.682124 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:11.682130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:11.682337 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 18:47:11.682484 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 18:47:11.682722 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":126,"threads_started":1}
I20260927 18:47:11.684566 4427 heartbeater.cc:347] Connected to a master server at 127.29.159.126:40221
I20260927 18:47:11.684767 4427 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:11.685166 4427 heartbeater.cc:511] Master 127.29.159.126:40221 requested a full tablet report, sending...
I20260927 18:47:11.686103 3968 ts_manager.cc:194] Registered new tserver with Master: 7fae20bb731f4ce2bea52eaa0418c7d1 (127.29.159.67:35793)
I20260927 18:47:11.686601 3968 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:47191
I20260927 18:47:11.688382 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.159.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.125.keytab.
Entry for principal oryx/127.29.159.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.125.keytab.
I20260927 18:47:11.715574 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.125:40555
--webserver_interface=127.29.159.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.125.keytab
--principal=oryx/127.29.159.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:40221,127.29.159.125:40555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:11.829012 4432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:11.829321 4432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:11.829409 4432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:11.833978 4432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:11.834064 4432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:11.834112 4432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:11.834170 4432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:11.834210 4432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:11.839471 4432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:40221,127.29.159.125:40555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.125.keytab
--principal=oryx/127.29.159.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.125:40555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.159.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
W20260927 18:47:11.839725 4432 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.29.159.126:40221,127.29.159.125:40555'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 18:47:11.840617 4432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:11.841657 4432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:11.850565 4440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:11.850602 4437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:11.850602 4438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:11.850926 4432 server_base.cc:1034] running on GCE node
I20260927 18:47:11.851564 4432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:11.852712 4432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:11.853874 4432 hybrid_clock.cc:648] HybridClock initialized: now 1790534831853860 us; error 29 us; skew 500 ppm
Sep 27 18:47:11 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534831, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:11.856645 4432 init.cc:377] Logged in from keytab as oryx/127.29.159.125@KRBTEST.COM (short username oryx)
I20260927 18:47:11.857493 4432 webserver.cc:460] Webserver started at http://127.29.159.125:41847/ using document root <none> and password file <none>
I20260927 18:47:11.858052 4432 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:11.858152 4432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:11.858378 4432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:11.860462 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data/instance:
uuid: "59f72f81f60a469c8b9fdae0433cc518"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "41848e1d1016290e205932f8cb43e264"
server_key_iv: "6a444b7d7d052245f94b6d1dc47cf98c"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.861022 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/wal/instance:
uuid: "59f72f81f60a469c8b9fdae0433cc518"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "41848e1d1016290e205932f8cb43e264"
server_key_iv: "6a444b7d7d052245f94b6d1dc47cf98c"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.864926 4432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:47:11.867219 4447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:11.868417 4432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:11.868562 4432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/wal
uuid: "59f72f81f60a469c8b9fdae0433cc518"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "41848e1d1016290e205932f8cb43e264"
server_key_iv: "6a444b7d7d052245f94b6d1dc47cf98c"
server_key_version: "encryptionkey@0"
I20260927 18:47:11.868669 4432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:11.892000 4432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:11.895185 4432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:11.895380 4432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:11.902186 4432 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.125:40555
I20260927 18:47:11.902479 4499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.125:40555 every 8 connection(s)
I20260927 18:47:11.903164 4432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
I20260927 18:47:11.905808 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:11.909372 4500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:47:11.911725 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 4432
Sep 27 18:47:11 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534831, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.125@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
Sep 27 18:47:11 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.159.125@KRBTEST.COM
I20260927 18:47:11.923359 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.911239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.29.159.126:40221)
0927 18:47:11.911376 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:11.911379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:11.912206 (+ 827us) server_negotiation.cc:407] Connection header received
0927 18:47:11.913239 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:11.913242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:11.913294 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:11.913381 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:11.914281 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.914854 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.916067 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.916426 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.916506 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.919313 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:11.919317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:11.919319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:11.919350 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:11.920974 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.921587 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.921590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.921591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.921668 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.922028 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.922032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.922034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.922324 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:11.922433 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:11.922990 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 18:47:11.923171 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:47:11.924225 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.912579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.29.159.125:40555)
0927 18:47:11.912915 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:11.912926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:11.912965 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:47:11.913121 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:11.913135 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:11.913386 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:11.913555 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:11.914392 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.915472 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.916409 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.917019 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.917135 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.919776 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:11.919784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:11.919797 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:11.919825 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:11.921700 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.922157 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.922161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.922163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.922226 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.922507 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.922510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.922512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.922779 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:11.922972 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:11.923216 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 18:47:11.923344 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":227,"spinlock_wait_cycles":768,"thread_start_us":78,"threads_started":1}
I20260927 18:47:11.924225 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.911547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:35894)
0927 18:47:11.912031 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:11.912089 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:11.913030 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:11.913435 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:11.913442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:11.913774 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:11.914127 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.914140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.914981 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.914985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.915951 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.915958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.916568 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.916572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.916861 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.917522 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:11.917537 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:11.919149 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:11.921088 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.921094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.921104 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.921469 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.921773 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.921778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.921784 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.921931 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.922425 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:11.922429 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:11.922718 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 18:47:11.922890 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 18:47:11.923213 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":62,"thread_start_us":105,"threads_started":1}
I20260927 18:47:11.925595 4500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:47:11 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534831, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.125@KRBTEST.COM for oryx/127.29.159.125@KRBTEST.COM
I20260927 18:47:11.933312 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.926095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:40555 (local address 127.0.0.1:39460)
0927 18:47:11.926305 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:11.926318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:11.926387 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:11.926637 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:11.926640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:11.926759 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:11.926839 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.926843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.927585 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.927590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.928153 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.928162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.928615 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.928618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.928703 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.929131 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:11.929144 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:11.930323 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:11.932054 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.932057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.932058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.932241 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.932499 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.932501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.932503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.932580 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.932918 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:11.932920 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:11.932992 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 18:47:11.933077 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:47:11.933186 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260927 18:47:11.933372 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.926213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.29.159.125:40555)
0927 18:47:11.926479 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:11.926481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:11.926488 (+ 7us) server_negotiation.cc:407] Connection header received
0927 18:47:11.926507 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:11.926509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:11.926532 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:11.926581 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:11.926935 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.927412 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.928265 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.928507 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.928620 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.930463 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:11.930469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:11.930472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:11.930500 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:11.931904 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.932328 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.932330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.932331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.932397 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.932683 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.932685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.932686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.932835 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:11.932904 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:11.933128 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 18:47:11.933236 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":223}
I20260927 18:47:11.936299 4500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: Bootstrap starting.
I20260927 18:47:11.938520 4500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:11.939251 4500 log.cc:824] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:11.941016 4500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: No bootstrap required, opened a new log
I20260927 18:47:11.945689 4500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } }
I20260927 18:47:11.945935 4500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:11.946000 4500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f72f81f60a469c8b9fdae0433cc518, State: Initialized, Role: FOLLOWER
I20260927 18:47:11.946487 4500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } }
I20260927 18:47:11.947005 4506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } } }
I20260927 18:47:11.947152 4506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:11.947945 4500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:47:11.953720 4517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:47:11.953806 4517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:47:11.954524 4432 master_runner.cc:428] Detected that this master 59f72f81f60a469c8b9fdae0433cc518 is joining an existing cluster
I20260927 18:47:11.954583 4432 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.159.125:40555
I20260927 18:47:11.965046 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.956551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:35896)
0927 18:47:11.956999 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:11.957013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:11.957121 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:11.957359 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:11.957362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:11.957577 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:11.957727 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.957733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.958693 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.958697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.959660 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.959671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.960267 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.960274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.960375 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.960706 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:11.960721 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:11.961305 (+ 584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:11.963344 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.963348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.963350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.963724 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.964004 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.964008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.964010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.964098 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.964476 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:11.964479 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:11.964581 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 18:47:11.964694 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 18:47:11.964822 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"spinlock_wait_cycles":4224,"thread_start_us":104,"threads_started":1}
I20260927 18:47:11.965523 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.956599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.29.159.126:40221)
0927 18:47:11.956722 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:11.956727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:11.957090 (+ 363us) server_negotiation.cc:407] Connection header received
0927 18:47:11.957209 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:11.957211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:11.957250 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:11.957344 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:11.957833 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.958569 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.959811 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.960147 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.960257 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.961449 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:11.961453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:11.961456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:11.961482 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:11.963228 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.963824 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.963827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.963829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.963915 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.964190 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.964194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.964196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.964382 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:11.964486 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:11.965256 (+ 770us) server_negotiation.cc:299] Negotiation successful
0927 18:47:11.965380 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:47:11.965927 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.956951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.29.159.125:40555)
0927 18:47:11.957082 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:11.957086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:11.957383 (+ 297us) server_negotiation.cc:407] Connection header received
0927 18:47:11.957530 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:11.957533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:11.957575 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:11.957656 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:11.958196 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.958842 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.959721 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.959937 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.960041 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.961789 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:11.961793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:11.961796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:11.961820 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:11.963723 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.964525 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.964528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.964530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.964606 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.964926 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.964929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.964931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.965143 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:11.965235 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:11.965666 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 18:47:11.965775 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":1280}
I20260927 18:47:11.966367 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.956610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:40555 (local address 127.0.0.1:39476)
0927 18:47:11.957316 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:11.957328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:11.957438 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:11.957731 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:11.957735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:11.957944 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:11.958059 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.958067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.958937 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.958943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.959613 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.959622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.960048 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.960051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.960132 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.960629 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:11.960650 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:11.961305 (+ 655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:11.963843 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.963846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.963848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.964086 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.964712 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.964715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.964717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.964817 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.965405 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:11.965408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:11.965498 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 18:47:11.966089 (+ 591us) client_negotiation.cc:240] Negotiation successful
0927 18:47:11.966204 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"spinlock_wait_cycles":3840,"thread_start_us":98,"threads_started":1}
Sep 27 18:47:11 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534830, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.126@KRBTEST.COM for oryx/127.29.159.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.29.159.124@KRBTEST.COM; defaulting to no policy
I20260927 18:47:11.986608 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.970003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.29.159.125:40555)
0927 18:47:11.970127 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:11.970131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:11.970496 (+ 365us) server_negotiation.cc:407] Connection header received
0927 18:47:11.970677 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:11.970680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:11.970728 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:11.970818 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:11.971483 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.972031 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.972959 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.973278 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.973393 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.982524 (+ 9131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:11.982529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:11.982532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:11.982559 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:11.984426 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.985040 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.985044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.985046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.985130 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:11.985468 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:11.985480 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:11.985482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:11.985743 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:11.985848 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:11.986294 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 18:47:11.986415 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:47:11.986642 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:11.969953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:40555 (local address 127.0.0.1:39482)
0927 18:47:11.970368 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:11.970416 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:11.970545 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:11.970854 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:11.970859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:11.971233 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:11.971335 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.971341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.972162 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.972165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.972853 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:11.972862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:11.973395 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:11.973398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:11.973520 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:11.980800 (+ 7280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:11.980813 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:11.982324 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:11.984560 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.984574 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.984582 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.984917 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.985235 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:11.985240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:11.985242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:11.985347 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:11.985876 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:11.985883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:11.986040 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 18:47:11.986221 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 18:47:11.986485 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":115,"threads_started":1}
I20260927 18:47:11.988358 3968 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.159.125:40555: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518"
member_type: NON_VOTER
last_known_addr {
host: "127.29.159.125"
port: 40555
}
attrs {
promote: true
}
}
dest_uuid: "1af467bdd4fc4d3f90cf79667730d4f3"
cas_config_opid_index: -1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.124@KRBTEST.COM" created.
I20260927 18:47:11.989159 3968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } }
I20260927 18:47:11.991248 4527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 59f72f81f60a469c8b9fdae0433cc518 (127.29.159.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } } }
I20260927 18:47:11.992700 4474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:11.993089 4527 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.159.125:40555
I20260927 18:47:11.993307 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } } }
I20260927 18:47:11.993408 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:11.993645 4432 master.cc:537] Master@127.29.159.125:40555 shutting down...
I20260927 18:47:11.994108 4474 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Refusing update from remote peer 1af467bdd4fc4d3f90cf79667730d4f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 18:47:11.994920 4528 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:47:11.997694 3952 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 18:47:11.998464 4432 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:11.998781 4432 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:11.998904 4432 tablet_replica.cc:330] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: stopping tablet replica
I20260927 18:47:12.004208 4432 master.cc:552] Master@127.29.159.125:40555 shutdown complete.
I20260927 18:47:12.004251 4432 master_runner.cc:305] Clearing existing system tablet
I20260927 18:47:12.004524 4432 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:12.004592 4432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:12.006194 4432 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 18:47:12.007236 4533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.007467 4432 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:12.007741 4432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:12.007794 4432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/wal
uuid: "59f72f81f60a469c8b9fdae0433cc518"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "41848e1d1016290e205932f8cb43e264"
server_key_iv: "6a444b7d7d052245f94b6d1dc47cf98c"
server_key_version: "encryptionkey@0"
I20260927 18:47:12.008427 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.008095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:40555 (local address 127.0.0.1:39484)
0927 18:47:12.008253 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.008335 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 18:47:12.008607 3952 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 -> Peer 59f72f81f60a469c8b9fdae0433cc518 (127.29.159.125:40555): Couldn't send request to peer 59f72f81f60a469c8b9fdae0433cc518. Status: Network error: Client connection negotiation failed: client connection to 127.29.159.125:40555: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:47:12.010049 4432 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.124.keytab.
Entry for principal oryx/127.29.159.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.124.keytab.
I20260927 18:47:12.013621 4432 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:47:12.013731 4432 log.cc:1196] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 18:47:12.014021 4432 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: Deleting consensus metadata
I20260927 18:47:12.014204 4432 master_runner.cc:315] Copying system tablet from 127.29.159.126:40221
I20260927 18:47:12.014647 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.124:44863
--webserver_interface=127.29.159.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.124.keytab
--principal=oryx/127.29.159.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:40221,127.29.159.125:40555,127.29.159.124:44863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:47:12.015561 4432 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: tablet copy: Beginning tablet copy session from remote peer at address 127.29.159.126:40221
I20260927 18:47:12.025820 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.016166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:35900)
0927 18:47:12.016618 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.016646 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.016785 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.017093 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.017098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.017328 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.017497 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.017508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.018688 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.018695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.019896 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.019913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.020507 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.020513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.020662 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.021284 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.021305 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.021815 (+ 510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.024089 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.024093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.024095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.024338 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.024670 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.024673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.024675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.024770 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.025270 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.025272 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.025367 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.025473 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.025606 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":84,"threads_started":1}
I20260927 18:47:12.025929 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.016283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.29.159.126:40221)
0927 18:47:12.016455 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.016461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.016756 (+ 295us) server_negotiation.cc:407] Connection header received
0927 18:47:12.016891 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.016895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.016949 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.017038 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.017682 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.018544 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.020031 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.020371 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.020489 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.021974 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.021978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.021981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.022008 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.023966 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.024448 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.024452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.024456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.024558 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.024873 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.024879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.024884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.025151 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.025282 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.025557 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.025703 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:47:12.026793 3988 tablet_copy_service.cc:140] P 1af467bdd4fc4d3f90cf79667730d4f3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 59f72f81f60a469c8b9fdae0433cc518 ({username='oryx', principal='oryx/127.29.159.125@KRBTEST.COM'} at 127.0.0.1:35900)
I20260927 18:47:12.026976 3988 tablet_copy_service.cc:161] P 1af467bdd4fc4d3f90cf79667730d4f3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 59f72f81f60a469c8b9fdae0433cc518 at {username='oryx', principal='oryx/127.29.159.125@KRBTEST.COM'} at 127.0.0.1:35900: session id = 59f72f81f60a469c8b9fdae0433cc518-00000000000000000000000000000000
I20260927 18:47:12.028858 3988 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:47:12.030629 4432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:47:12.033310 4432 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: tablet copy: Starting download of 0 data blocks...
I20260927 18:47:12.033504 4432 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: tablet copy: Starting download of 1 WAL segments...
I20260927 18:47:12.035259 4432 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:47:12.037108 3988 tablet_copy_service.cc:342] P 1af467bdd4fc4d3f90cf79667730d4f3: Request end of tablet copy session 59f72f81f60a469c8b9fdae0433cc518-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.159.125@KRBTEST.COM'} at 127.0.0.1:35900
I20260927 18:47:12.037247 3988 tablet_copy_service.cc:433] P 1af467bdd4fc4d3f90cf79667730d4f3: ending tablet copy session 59f72f81f60a469c8b9fdae0433cc518-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 59f72f81f60a469c8b9fdae0433cc518
I20260927 18:47:12.044462 4432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:12.046144 4541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.046245 4544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.046509 4542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:12.047155 4432 server_base.cc:1034] running on GCE node
I20260927 18:47:12.047308 4432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:12.047628 4432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:12.048785 4432 hybrid_clock.cc:648] HybridClock initialized: now 1790534832048771 us; error 29 us; skew 500 ppm
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:12.050369 4432 init.cc:377] Logged in from keytab as oryx/127.29.159.125@KRBTEST.COM (short username oryx)
I20260927 18:47:12.050854 4432 webserver.cc:460] Webserver started at http://127.29.159.125:42025/ using document root <none> and password file <none>
I20260927 18:47:12.051035 4432 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:12.051084 4432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:12.052416 4432 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:47:12.053465 4551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.053818 4432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 18:47:12.053889 4432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/wal
uuid: "59f72f81f60a469c8b9fdae0433cc518"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "41848e1d1016290e205932f8cb43e264"
server_key_iv: "6a444b7d7d052245f94b6d1dc47cf98c"
server_key_version: "encryptionkey@0"
I20260927 18:47:12.053962 4432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:12.069702 4432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:12.072080 4432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:12.076778 4432 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.125:40555
I20260927 18:47:12.076818 4603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.125:40555 every 8 connection(s)
I20260927 18:47:12.077493 4432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
I20260927 18:47:12.079504 4604 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:47:12.080660 4604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: Bootstrap starting.
I20260927 18:47:12.091279 4604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:47:12.091641 4604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: Bootstrap complete.
I20260927 18:47:12.092289 4604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } }
I20260927 18:47:12.092446 4604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 59f72f81f60a469c8b9fdae0433cc518, State: Initialized, Role: LEARNER
I20260927 18:47:12.092571 4604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } }
I20260927 18:47:12.092941 4607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } } }
I20260927 18:47:12.093061 4604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:12.093143 4607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:47:12.096439 4618 catalog_manager.cc:1259] Loaded cluster ID: 3cd05ce26e8f4e80ac24854feb0b86a2
I20260927 18:47:12.096549 4618 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: loading cluster ID for follower catalog manager: success
I20260927 18:47:12.098644 4618 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: acquiring CA information for follower catalog manager: success
I20260927 18:47:12.100083 4618 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 18:47:12.146586 4539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:12.146834 4539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:12.146873 4539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:12.151065 4539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:12.151127 4539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:12.151147 4539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:12.151163 4539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:12.151181 4539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:12.156651 4539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:40221,127.29.159.125:40555,127.29.159.124:44863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.124.keytab
--principal=oryx/127.29.159.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.124:44863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.159.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:12.157689 4539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:12.158725 4539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:12.167212 4623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.167335 4624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.167224 4626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:12.167439 4539 server_base.cc:1034] running on GCE node
I20260927 18:47:12.168092 4539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:12.169159 4539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:12.170346 4539 hybrid_clock.cc:648] HybridClock initialized: now 1790534832170330 us; error 27 us; skew 500 ppm
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:12.173381 4539 init.cc:377] Logged in from keytab as oryx/127.29.159.124@KRBTEST.COM (short username oryx)
I20260927 18:47:12.174242 4539 webserver.cc:460] Webserver started at http://127.29.159.124:36201/ using document root <none> and password file <none>
I20260927 18:47:12.174813 4539 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:12.174907 4539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:12.175140 4539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:12.177232 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data/instance:
uuid: "64bf2bba135d41f28b57fd32fc968054"
format_stamp: "Formatted at 2026-09-27 18:47:12 on dist-test-slave-8tg2"
server_key: "a1934d2dcb0c103a1a9aa8241867c254"
server_key_iv: "9cee7126a98fc91ac3d9eef3e077bfe0"
server_key_version: "encryptionkey@0"
I20260927 18:47:12.177824 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/wal/instance:
uuid: "64bf2bba135d41f28b57fd32fc968054"
format_stamp: "Formatted at 2026-09-27 18:47:12 on dist-test-slave-8tg2"
server_key: "a1934d2dcb0c103a1a9aa8241867c254"
server_key_iv: "9cee7126a98fc91ac3d9eef3e077bfe0"
server_key_version: "encryptionkey@0"
I20260927 18:47:12.181618 4539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:47:12.184099 4633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.185405 4539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:12.185545 4539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/wal
uuid: "64bf2bba135d41f28b57fd32fc968054"
format_stamp: "Formatted at 2026-09-27 18:47:12 on dist-test-slave-8tg2"
server_key: "a1934d2dcb0c103a1a9aa8241867c254"
server_key_iv: "9cee7126a98fc91ac3d9eef3e077bfe0"
server_key_version: "encryptionkey@0"
I20260927 18:47:12.185660 4539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:12.208948 4539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:12.212451 4539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:12.212621 4539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:12.219480 4539 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.124:44863
I20260927 18:47:12.219792 4685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.124:44863 every 8 connection(s)
I20260927 18:47:12.220549 4539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
I20260927 18:47:12.222018 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 4539
I20260927 18:47:12.224051 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:12.228240 4686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.159.124@KRBTEST.COM
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.124@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:12.237330 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.223485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.29.159.124:44863)
0927 18:47:12.223923 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.223944 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.224134 (+ 190us) server_negotiation.cc:407] Connection header received
0927 18:47:12.224912 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.224928 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.225238 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.225580 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.226359 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.227618 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.228818 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.229272 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.229389 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.232433 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.232437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.232446 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.232472 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.234329 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.234779 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.234785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.234792 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.234905 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.235197 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.235201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.235202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.235638 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.235875 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.236089 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.236384 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
I20260927 18:47:12.239940 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.230240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.29.159.126:40221)
0927 18:47:12.230411 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.230416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.230666 (+ 250us) server_negotiation.cc:407] Connection header received
0927 18:47:12.230829 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.230832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.230886 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.230973 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.231677 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.232265 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.233738 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.233961 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.234072 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.236358 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.236361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.236363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.236385 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.237972 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.238491 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.238496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.238498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.238576 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.238875 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.238880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.238882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.239050 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.239160 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.239636 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.239793 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:47:12.239951 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.230125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:35908)
0927 18:47:12.230554 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.230591 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.230724 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.231018 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.231023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.231359 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.231558 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.231564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.232781 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.232785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.233619 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.233628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.234071 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.234074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.234186 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.234665 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.234676 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.236165 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.238098 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.238108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.238175 (+ 67us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.238385 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.238698 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.238701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.238702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.238786 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.239272 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.239285 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.239375 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.239570 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.239745 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":119,"threads_started":1}
I20260927 18:47:12.241034 4686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:47:12.245225 4159 heartbeater.cc:503] Master 127.29.159.126:40221 was elected leader, sending a full tablet report...
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.124@KRBTEST.COM for oryx/127.29.159.125@KRBTEST.COM
I20260927 18:47:12.249706 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.241483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:40555 (local address 127.0.0.1:39490)
0927 18:47:12.241600 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.241612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.241672 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.242068 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.242072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.242278 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.242372 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.242377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.243268 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.243272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.243878 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.243886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.244368 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.244372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.244466 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.244856 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.244866 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.246186 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.248227 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.248231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.248233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.248432 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.248716 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.248720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.248722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.248807 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.249183 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.249187 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.249293 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.249400 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.249540 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 18:47:12.249738 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.241642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.29.159.125:40555)
0927 18:47:12.241871 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.241874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.241885 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:47:12.241918 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.241920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.241956 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.242057 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.242490 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.243152 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.243989 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.244223 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.244363 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.246359 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.246362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.246364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.246386 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.248114 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.248534 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.248538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.248540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.248623 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.248897 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.248902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.248903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.249086 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.249193 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.249466 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.249588 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260927 18:47:12.250661 4686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.124@KRBTEST.COM for oryx/127.29.159.124@KRBTEST.COM
I20260927 18:47:12.258603 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.251149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:44863 (local address 127.0.0.1:53756)
0927 18:47:12.251271 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.251286 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.251361 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.251685 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.251687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.251814 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.251883 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.251887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.252966 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.252969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.253461 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.253467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.253934 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.253936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.254055 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.254468 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.254478 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.255775 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.257231 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.257234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.257235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.257416 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.257652 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.257654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.257655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.257723 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.258104 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.258106 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.258205 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.258313 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.258438 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":23}
I20260927 18:47:12.258616 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.251307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.29.159.124:44863)
0927 18:47:12.251486 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.251491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.251509 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:47:12.251537 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.251539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.251573 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.251659 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.251982 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.252859 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.253577 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.253822 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.253952 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.255893 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.255896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.255898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.255918 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.257131 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.257495 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.257498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.257499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.257563 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.257795 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.257797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.257799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.257958 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.258032 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.258362 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.258491 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":50}
I20260927 18:47:12.261747 4686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: Bootstrap starting.
I20260927 18:47:12.264313 4686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:12.265070 4686 log.cc:824] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:12.267045 4686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: No bootstrap required, opened a new log
I20260927 18:47:12.270082 4686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } }
I20260927 18:47:12.270303 4686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:12.270364 4686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64bf2bba135d41f28b57fd32fc968054, State: Initialized, Role: FOLLOWER
I20260927 18:47:12.270828 4686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } }
I20260927 18:47:12.271330 4692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } } }
I20260927 18:47:12.271500 4692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:12.272104 4686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:47:12.276826 4703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:47:12.276914 4703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:47:12.280133 4539 master_runner.cc:428] Detected that this master 64bf2bba135d41f28b57fd32fc968054 is joining an existing cluster
I20260927 18:47:12.280202 4539 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.159.124:44863
I20260927 18:47:12.290467 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 4030
I20260927 18:47:12.297525 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:45531
--local_ip_for_outbound_sockets=127.29.159.65
--tserver_master_addrs=127.29.159.126:40221
--webserver_port=35513
--webserver_interface=127.29.159.65
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab
--principal=oryx/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.159.126:40221,127.29.159.125:40555
--tserver_master_addrs=127.29.159.126:40221,127.29.159.125:40555,127.29.159.124:44863 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:47:12.309813 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.282838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:35912)
0927 18:47:12.283353 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.283367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.283520 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.283906 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.283909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.284168 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.284320 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.284327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.285368 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.285371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.286205 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.286216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.286883 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.286886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.287001 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.287495 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.287508 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.304128 (+ 16620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.307305 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.307309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.307311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.307642 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.308657 (+ 1015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.308660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.308662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.308771 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.309236 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.309239 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.309364 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.309460 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.309614 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":44,"thread_start_us":116,"threads_started":1}
I20260927 18:47:12.309826 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.282415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.29.159.125:40555)
0927 18:47:12.282629 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.282633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.282781 (+ 148us) server_negotiation.cc:407] Connection header received
0927 18:47:12.282996 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.283002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.283060 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.283116 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.283836 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.284635 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.285839 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.286190 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.286280 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.304974 (+ 18694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.304982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.304989 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.305029 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.306926 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.307658 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.307664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.307669 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.307772 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.308094 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.308100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.308104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.308356 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.308437 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.309555 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.309680 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260927 18:47:12.310614 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.282378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.29.159.126:40221)
0927 18:47:12.282503 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.282508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.283456 (+ 948us) server_negotiation.cc:407] Connection header received
0927 18:47:12.283634 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.283637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.283694 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.283759 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.284453 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.285235 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.286332 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.286656 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.286736 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.304356 (+ 17620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.304360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.304362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.304390 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.307122 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.308438 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.308441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.308444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.308544 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.308876 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.308882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.308885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.309106 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.309195 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.310329 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.310445 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:47:12.311358 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.283145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.29.159.124:44863)
0927 18:47:12.283293 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.283297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.283523 (+ 226us) server_negotiation.cc:407] Connection header received
0927 18:47:12.283712 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.283715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.283753 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.283798 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.284461 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.285190 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.286609 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.287293 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.287366 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.304355 (+ 16989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.304359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.304362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.304389 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.308308 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.308831 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.308836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.308841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.308913 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.309173 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.309175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.309177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.310106 (+ 929us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.310206 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.311115 (+ 909us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.311222 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":690,"server-negotiator.queue_time_us":58}
I20260927 18:47:12.312059 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.282866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:44863 (local address 127.0.0.1:53770)
0927 18:47:12.283443 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.283460 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.283613 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.283912 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.283915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.284189 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.284341 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.284349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.285324 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.285327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.286485 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.286495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.287519 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.287523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.287607 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.288040 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.288054 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.304231 (+ 16177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.308477 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.308483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.308487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.308721 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.309008 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.309010 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.309012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.309080 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.310618 (+ 1538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.310624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.310952 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.311775 (+ 823us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.311904 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":1,"spinlock_wait_cycles":2432,"thread_start_us":98,"threads_started":1}
I20260927 18:47:12.313645 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.282386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:40555 (local address 127.0.0.1:39496)
0927 18:47:12.282681 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.282697 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.282855 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.283279 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.283282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.283540 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.283707 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.283715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.284766 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.284769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.285715 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.285723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.286451 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.286454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.286539 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.287065 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.287082 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.304788 (+ 17706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.307073 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.307077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.307080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.307532 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.307894 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.307899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.307901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.307982 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.309303 (+ 1321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.309306 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.309374 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.313292 (+ 3918us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.313433 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1,"spinlock_wait_cycles":1920,"thread_start_us":128,"threads_started":1}
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534830, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.126@KRBTEST.COM for oryx/127.29.159.124@KRBTEST.COM
I20260927 18:47:12.321969 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.314838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.29.159.124:44863)
0927 18:47:12.314944 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.314947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.314957 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:47:12.315007 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.315009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.315046 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.315142 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.315618 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.316089 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.316872 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.317082 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.317188 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.319078 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.319087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.319089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.319108 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.320459 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.320869 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.320875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.320877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.320949 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.321214 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.321219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.321221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.321380 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.321481 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.321732 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.321836 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:47:12.321969 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.314627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:44863 (local address 127.0.0.1:53774)
0927 18:47:12.314766 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.314782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.314866 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.315162 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.315166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.315347 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.315487 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.315499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.316198 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.316205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.316736 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.316744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.317188 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.317192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.317300 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.317741 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.317758 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.318949 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.320577 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.320580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.320582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.320774 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.321039 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.321043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.321045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.321127 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.321473 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.321476 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.321573 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.321683 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.321820 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 18:47:12.323705 3968 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.159.124:44863: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "64bf2bba135d41f28b57fd32fc968054"
member_type: NON_VOTER
last_known_addr {
host: "127.29.159.124"
port: 44863
}
attrs {
promote: true
}
}
dest_uuid: "1af467bdd4fc4d3f90cf79667730d4f3"
cas_config_opid_index: 5
I20260927 18:47:12.324260 3968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } }
I20260927 18:47:12.325372 4528 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 64bf2bba135d41f28b57fd32fc968054 (127.29.159.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.326650 4528 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.159.124:44863
I20260927 18:47:12.326817 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.326937 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:12.327517 4539 master.cc:537] Master@127.29.159.124:44863 shutting down...
I20260927 18:47:12.327720 4660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:12.329353 4660 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 FOLLOWER]: Refusing update from remote peer 1af467bdd4fc4d3f90cf79667730d4f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 18:47:12.330034 4527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:47:12.332068 3954 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 -> Peer 64bf2bba135d41f28b57fd32fc968054 (127.29.159.124:44863): Couldn't send request to peer 64bf2bba135d41f28b57fd32fc968054. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 18:47:12.334128 4539 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:12.334405 4539 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:12.334497 4539 tablet_replica.cc:330] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: stopping tablet replica
I20260927 18:47:12.337738 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.325486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:40555 (local address 127.0.0.1:39508)
0927 18:47:12.332968 (+ 7482us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.332986 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.333077 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.333421 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.333424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.333441 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:12.333538 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.333544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.334702 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.334705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.336036 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.336051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.337270 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.337277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.337376 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.337442 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.337514 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.337601 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7401}
I20260927 18:47:12.338209 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.325708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39508 (local address 127.29.159.125:40555)
0927 18:47:12.325902 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.325906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.333169 (+ 7263us) server_negotiation.cc:407] Connection header received
0927 18:47:12.333225 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.333228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.333291 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.333386 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:12.333671 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.334479 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.336194 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.336913 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.337017 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.337162 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.337988 (+ 826us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.338076 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 18:47:12.339006 4578 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 LEARNER]: Refusing update from remote peer 1af467bdd4fc4d3f90cf79667730d4f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 18:47:12.339623 4527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 18:47:12.341419 4578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 64bf2bba135d41f28b57fd32fc968054 (127.29.159.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.343334 4607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af467bdd4fc4d3f90cf79667730d4f3. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.343514 4607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:47:12.343659 4607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: NON_VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: true } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.343746 4607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:47:12.344410 4527 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3: attempting to promote NON_VOTER 59f72f81f60a469c8b9fdae0433cc518 to VOTER
I20260927 18:47:12.344524 4539 master.cc:552] Master@127.29.159.124:44863 shutdown complete.
I20260927 18:47:12.344561 4539 master_runner.cc:305] Clearing existing system tablet
I20260927 18:47:12.344795 4539 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:12.344847 4539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:12.344947 4527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } }
I20260927 18:47:12.346237 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.345938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:44863 (local address 127.0.0.1:53788)
0927 18:47:12.346087 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.346152 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 18:47:12.346848 4578 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 LEARNER]: Refusing update from remote peer 1af467bdd4fc4d3f90cf79667730d4f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 18:47:12.347024 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.346711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:44863 (local address 127.0.0.1:53790)
0927 18:47:12.346875 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.346963 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260927 18:47:12.347157 3954 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 -> Peer 64bf2bba135d41f28b57fd32fc968054 (127.29.159.124:44863): Couldn't send request to peer 64bf2bba135d41f28b57fd32fc968054. Status: Network error: Client connection negotiation failed: client connection to 127.29.159.124:44863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:47:12.347409 4528 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 18:47:12.347877 4539 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:12.349059 4717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.349329 4528 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 59f72f81f60a469c8b9fdae0433cc518 (127.29.159.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.349915 4577 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 59f72f81f60a469c8b9fdae0433cc518 (127.29.159.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.350970 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.351060 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:12.351207 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.351303 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:12.351508 4607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.351601 4607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:12.352342 4539 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:12.352489 4539 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 18:47:12.352557 4539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/wal
uuid: "64bf2bba135d41f28b57fd32fc968054"
format_stamp: "Formatted at 2026-09-27 18:47:12 on dist-test-slave-8tg2"
server_key: "a1934d2dcb0c103a1a9aa8241867c254"
server_key_iv: "9cee7126a98fc91ac3d9eef3e077bfe0"
server_key_version: "encryptionkey@0"
I20260927 18:47:12.354323 4539 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:12.357106 4539 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:47:12.357227 4539 log.cc:1196] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 18:47:12.357483 4539 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: Deleting consensus metadata
I20260927 18:47:12.357667 4539 master_runner.cc:315] Copying system tablet from 127.29.159.126:40221
I20260927 18:47:12.358789 4539 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: tablet copy: Beginning tablet copy session from remote peer at address 127.29.159.126:40221
I20260927 18:47:12.373720 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.359391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.29.159.126:40221)
0927 18:47:12.359528 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.359532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.366767 (+ 7235us) server_negotiation.cc:407] Connection header received
0927 18:47:12.366948 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.366951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.367006 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.367092 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.367600 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.368123 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.368959 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.369183 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.369305 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.370430 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.370433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.370436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.370461 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.371958 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.372409 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.372413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.372415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.372496 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.372801 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.372806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.372808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.372981 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.373094 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.373448 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.373554 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:47:12.373778 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.359338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:35928)
0927 18:47:12.366650 (+ 7312us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.366665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.366832 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.367116 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.367118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.367324 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.367477 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.367487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.368235 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.368240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.368832 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.368839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.369312 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.369315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.369409 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.369967 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.369978 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.370284 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.372060 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.372063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.372066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.372307 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.372586 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.372591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.372593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.372677 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.373142 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.373187 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.373283 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.373394 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.373532 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7240,"thread_start_us":104,"threads_started":1}
I20260927 18:47:12.374603 3988 tablet_copy_service.cc:140] P 1af467bdd4fc4d3f90cf79667730d4f3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 64bf2bba135d41f28b57fd32fc968054 ({username='oryx', principal='oryx/127.29.159.124@KRBTEST.COM'} at 127.0.0.1:35928)
I20260927 18:47:12.374717 3988 tablet_copy_service.cc:161] P 1af467bdd4fc4d3f90cf79667730d4f3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 64bf2bba135d41f28b57fd32fc968054 at {username='oryx', principal='oryx/127.29.159.124@KRBTEST.COM'} at 127.0.0.1:35928: session id = 64bf2bba135d41f28b57fd32fc968054-00000000000000000000000000000000
I20260927 18:47:12.375648 3988 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:47:12.376814 4539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:47:12.378602 4539 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: tablet copy: Starting download of 0 data blocks...
I20260927 18:47:12.378793 4539 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: tablet copy: Starting download of 1 WAL segments...
I20260927 18:47:12.380357 4539 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:47:12.382012 3988 tablet_copy_service.cc:342] P 1af467bdd4fc4d3f90cf79667730d4f3: Request end of tablet copy session 64bf2bba135d41f28b57fd32fc968054-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.159.124@KRBTEST.COM'} at 127.0.0.1:35928
I20260927 18:47:12.382103 3988 tablet_copy_service.cc:433] P 1af467bdd4fc4d3f90cf79667730d4f3: ending tablet copy session 64bf2bba135d41f28b57fd32fc968054-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 64bf2bba135d41f28b57fd32fc968054
I20260927 18:47:12.388459 4539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:12.390146 4725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.390301 4726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.390540 4728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:12.391003 4539 server_base.cc:1034] running on GCE node
I20260927 18:47:12.391355 4539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:12.391937 4539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:12.393435 4539 hybrid_clock.cc:648] HybridClock initialized: now 1790534832393422 us; error 33 us; skew 500 ppm
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:12.395356 4539 init.cc:377] Logged in from keytab as oryx/127.29.159.124@KRBTEST.COM (short username oryx)
I20260927 18:47:12.395897 4539 webserver.cc:460] Webserver started at http://127.29.159.124:43907/ using document root <none> and password file <none>
I20260927 18:47:12.396090 4539 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:12.396147 4539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:12.397307 4539 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:12.398288 4735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.398514 4539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:12.398586 4539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/wal
uuid: "64bf2bba135d41f28b57fd32fc968054"
format_stamp: "Formatted at 2026-09-27 18:47:12 on dist-test-slave-8tg2"
server_key: "a1934d2dcb0c103a1a9aa8241867c254"
server_key_iv: "9cee7126a98fc91ac3d9eef3e077bfe0"
server_key_version: "encryptionkey@0"
I20260927 18:47:12.398674 4539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:12.430378 4539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:12.432610 4539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:12.437523 4539 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.124:44863
I20260927 18:47:12.438019 4539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
I20260927 18:47:12.439409 4788 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:47:12.440459 4788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: Bootstrap starting.
I20260927 18:47:12.443014 4787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.124:44863 every 8 connection(s)
I20260927 18:47:12.449275 4788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:47:12.449585 4788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: Bootstrap complete.
I20260927 18:47:12.450069 4788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } }
I20260927 18:47:12.450207 4788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64bf2bba135d41f28b57fd32fc968054, State: Initialized, Role: LEARNER
I20260927 18:47:12.450344 4788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } }
I20260927 18:47:12.450680 4791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.450843 4788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:12.450855 4791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:47:12.454329 4802 catalog_manager.cc:1259] Loaded cluster ID: 3cd05ce26e8f4e80ac24854feb0b86a2
I20260927 18:47:12.454453 4802 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: loading cluster ID for follower catalog manager: success
I20260927 18:47:12.456164 4802 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: acquiring CA information for follower catalog manager: success
I20260927 18:47:12.457257 4802 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 18:47:12.466166 4712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:12.466552 4712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:12.466627 4712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:12.473465 4712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:12.473562 4712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:12.473662 4712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:12.482664 4712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab
--principal=oryx/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:45531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=35513
--tserver_master_addrs=127.29.159.126:40221,127.29.159.125:40555,127.29.159.124:44863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:12.484171 4712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:12.485297 4293 heartbeater.cc:503] Master 127.29.159.126:40221 was elected leader, sending a full tablet report...
I20260927 18:47:12.485651 4712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:12.499003 4808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.498981 4807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.499233 4810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:12.499332 4712 server_base.cc:1034] running on GCE node
I20260927 18:47:12.499886 4712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:12.500659 4712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:12.501863 4712 hybrid_clock.cc:648] HybridClock initialized: now 1790534832501830 us; error 49 us; skew 500 ppm
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:12.504985 4712 init.cc:377] Logged in from keytab as oryx/127.29.159.65@KRBTEST.COM (short username oryx)
I20260927 18:47:12.505970 4712 webserver.cc:460] Webserver started at http://127.29.159.65:35513/ using document root <none> and password file <none>
I20260927 18:47:12.506579 4712 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:12.506702 4712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:12.510859 4712 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:12.513089 4817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.514398 4712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:47:12.514562 4712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "cf85e20a1ec8466ca9bcd09b7df90878"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "8e36ef3de2464270b77f4e9404109864"
server_key_iv: "fa47468b341595f9533225219b4c9a0a"
server_key_version: "encryptionkey@0"
I20260927 18:47:12.515136 4712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:12.556047 4712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:12.559509 4712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:12.559739 4712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:12.560400 4712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:12.561372 4712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:12.561450 4712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.561528 4712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:12.561589 4712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.571336 4712 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:45531
I20260927 18:47:12.571358 4930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:45531 every 8 connection(s)
I20260927 18:47:12.572510 4712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:12.580024 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 4712
I20260927 18:47:12.580232 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 4164
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.125@KRBTEST.COM
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.124@KRBTEST.COM
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:12.589347 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:43979
--local_ip_for_outbound_sockets=127.29.159.66
--tserver_master_addrs=127.29.159.126:40221
--webserver_port=46295
--webserver_interface=127.29.159.66
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab
--principal=oryx/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.159.126:40221,127.29.159.125:40555
--tserver_master_addrs=127.29.159.126:40221,127.29.159.125:40555,127.29.159.124:44863 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:47:12.591751 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.574478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:52847 (local address 127.29.159.126:40221)
0927 18:47:12.574624 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.574629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.575970 (+ 1341us) server_negotiation.cc:407] Connection header received
0927 18:47:12.577163 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.577166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.577211 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.577307 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.578829 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.579669 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.582306 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.582669 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.582757 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.587141 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.587145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.587150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.587189 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.589189 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.589726 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.589730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.589732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.589816 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.590152 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.590156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.590158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.590375 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.590498 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.591353 (+ 855us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.591525 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:47:12.591799 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.575106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:55797 (local address 127.29.159.125:40555)
0927 18:47:12.575249 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.575254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.576203 (+ 949us) server_negotiation.cc:407] Connection header received
0927 18:47:12.577431 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.577434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.577485 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.577568 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.578863 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.579529 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.581665 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.582097 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.582192 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.586012 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.586018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.586025 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.586060 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.588658 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.589728 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.589734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.589739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.589845 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.590163 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.590169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.590174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.590441 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.590519 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.591339 (+ 820us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.591547 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:47:12.591823 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.574465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:33159 (local address 127.29.159.124:44863)
0927 18:47:12.574786 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.574791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.575837 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 18:47:12.577065 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.577070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.577153 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.577266 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.578885 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.579954 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.581664 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.582070 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.582155 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.586530 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.586534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.586537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.586570 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.588705 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.589482 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.589488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.589490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.589581 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.590032 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.590035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.590037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.590380 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.590475 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.591511 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.591659 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":88,"threads_started":1}
I20260927 18:47:12.593657 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.574906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:44863 (local address 127.29.159.65:33159)
0927 18:47:12.575658 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.575727 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.576838 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.577454 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.577459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.578127 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.578693 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.578702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.580064 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.580068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.581502 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.581514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.582369 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.582373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.582589 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.583461 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.583491 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.586350 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.588850 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.588855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.588858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.589356 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.589710 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.589713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.589715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.589912 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.590575 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.590583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.591034 (+ 451us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.592008 (+ 974us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.592491 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":456,"spinlock_wait_cycles":51072,"thread_start_us":143,"threads_started":1}
I20260927 18:47:12.593670 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.574906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.65:52847)
0927 18:47:12.575892 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.575906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.576824 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.577452 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.577460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.578146 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.578683 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.578697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.579780 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.579782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.582160 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.582169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.582955 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.582963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.583068 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.583576 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.583589 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.586940 (+ 3351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.589329 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.589333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.589336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.589590 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.589937 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.589944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.589949 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.590040 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.590575 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.590583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.591014 (+ 431us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.591946 (+ 932us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.592438 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":1423,"spinlock_wait_cycles":38400,"thread_start_us":167,"threads_started":1}
I20260927 18:47:12.594503 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.575014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:40555 (local address 127.29.159.65:55797)
0927 18:47:12.576090 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.576102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.576824 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.577607 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.577613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.578127 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.578693 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.578702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.579688 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.579692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.581516 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.581530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.582370 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.582376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.582593 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.583742 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.583773 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.585748 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.588802 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.588811 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.588832 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.589352 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.589966 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.589972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.589977 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.590066 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.590611 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.590616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.591011 (+ 395us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.592702 (+ 1691us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.592885 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":558,"spinlock_wait_cycles":31744,"thread_start_us":679,"threads_started":1}
I20260927 18:47:12.595036 4931 heartbeater.cc:347] Connected to a master server at 127.29.159.124:44863
I20260927 18:47:12.595072 4932 heartbeater.cc:347] Connected to a master server at 127.29.159.126:40221
I20260927 18:47:12.595314 4931 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:12.595314 4932 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:12.595662 4933 heartbeater.cc:347] Connected to a master server at 127.29.159.125:40555
I20260927 18:47:12.595738 4933 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:12.596024 4931 heartbeater.cc:511] Master 127.29.159.124:44863 requested a full tablet report, sending...
I20260927 18:47:12.596026 4932 heartbeater.cc:511] Master 127.29.159.126:40221 requested a full tablet report, sending...
I20260927 18:47:12.596029 4933 heartbeater.cc:511] Master 127.29.159.125:40555 requested a full tablet report, sending...
I20260927 18:47:12.597411 3968 ts_manager.cc:194] Re-registered known tserver with Master: cf85e20a1ec8466ca9bcd09b7df90878 (127.29.159.65:45531)
I20260927 18:47:12.597674 4568 ts_manager.cc:194] Registered new tserver with Master: cf85e20a1ec8466ca9bcd09b7df90878 (127.29.159.65:45531)
I20260927 18:47:12.597725 4752 ts_manager.cc:194] Registered new tserver with Master: cf85e20a1ec8466ca9bcd09b7df90878 (127.29.159.65:45531)
I20260927 18:47:12.598234 3968 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:52847
I20260927 18:47:12.688895 4427 heartbeater.cc:503] Master 127.29.159.126:40221 was elected leader, sending a full tablet report...
W20260927 18:47:12.714350 4939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:12.714644 4939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:12.714716 4939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:12.721388 4939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:12.721493 4939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:12.721606 4939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:12.727977 4939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab
--principal=oryx/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:43979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=46295
--tserver_master_addrs=127.29.159.126:40221,127.29.159.125:40555,127.29.159.124:44863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:12.729220 4939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:12.730304 4939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:12.740456 4947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.740469 4945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.740483 4944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:12.740803 4939 server_base.cc:1034] running on GCE node
I20260927 18:47:12.741241 4939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:12.741830 4939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:12.742988 4939 hybrid_clock.cc:648] HybridClock initialized: now 1790534832742945 us; error 54 us; skew 500 ppm
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:12.745816 4939 init.cc:377] Logged in from keytab as oryx/127.29.159.66@KRBTEST.COM (short username oryx)
I20260927 18:47:12.746695 4939 webserver.cc:460] Webserver started at http://127.29.159.66:46295/ using document root <none> and password file <none>
I20260927 18:47:12.747247 4939 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:12.747344 4939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:12.751241 4939 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:47:12.753468 4954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.754843 4939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:12.754935 4939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "d4346c5843c646e4bbf1b621a5d0c875"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "77ae633b01834423c54c945afa5d5d6e"
server_key_iv: "9a566938b3ba05b9fbec16d9ed03f3db"
server_key_version: "encryptionkey@0"
I20260927 18:47:12.755542 4939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:12.773046 4939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:12.776163 4939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:12.776320 4939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:12.776904 4939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:12.777711 4939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:12.777765 4939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.777805 4939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:12.777827 4939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.786293 4939 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:43979
I20260927 18:47:12.786335 5067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:43979 every 8 connection(s)
I20260927 18:47:12.787276 4939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:12.797187 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 4939
I20260927 18:47:12.797381 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 4298
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.124@KRBTEST.COM
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.125@KRBTEST.COM
I20260927 18:47:12.804121 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.789189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:34419 (local address 127.29.159.126:40221)
0927 18:47:12.789412 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.789416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.790258 (+ 842us) server_negotiation.cc:407] Connection header received
0927 18:47:12.791465 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.791469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.791525 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.791580 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.793043 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.793858 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.795155 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.795501 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.795594 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.799236 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.799240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.799243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.799273 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.801235 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.801979 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.801983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.801985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.802184 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.802603 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.802606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.802608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.803031 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.803133 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.803745 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.803905 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260927 18:47:12.804137 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.789057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:52997 (local address 127.29.159.125:40555)
0927 18:47:12.789320 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.789325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.790460 (+ 1135us) server_negotiation.cc:407] Connection header received
0927 18:47:12.791465 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.791469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.791525 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.791580 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.793043 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.793849 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.795180 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.795414 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.795584 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.799740 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.799745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.799749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.799780 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.801623 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.802240 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.802247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.802251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.802335 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.802636 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.802642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.802647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.802901 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.802973 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.803736 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.803877 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260927 18:47:12.804591 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.789746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:50627 (local address 127.29.159.124:44863)
0927 18:47:12.789880 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.789885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.790659 (+ 774us) server_negotiation.cc:407] Connection header received
0927 18:47:12.791451 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.791456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.791522 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.791623 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:12.793020 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.793549 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.794887 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.795232 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.795332 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.799635 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:12.799639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:12.799641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:12.799675 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:12.801364 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.801979 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.801983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.801985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.802080 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:12.802603 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:12.802606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:12.802608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:12.802820 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:12.802878 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.804302 (+ 1424us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.804435 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:47:12.805781 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.789520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:40555 (local address 127.29.159.66:52997)
0927 18:47:12.790385 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.790397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.791267 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.791794 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.791799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.792249 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.792875 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.792894 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.793992 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.793995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.795036 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.795047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.795582 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.795586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.795751 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.796568 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.796590 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.799569 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.801826 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.801832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.801837 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.802125 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.802446 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.802451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.802457 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.802542 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.803057 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.803066 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.803418 (+ 352us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.805160 (+ 1742us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.805332 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":135,"spinlock_wait_cycles":105728,"thread_start_us":147,"threads_started":1}
I20260927 18:47:12.805785 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.789520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:44863 (local address 127.29.159.66:50627)
0927 18:47:12.790559 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.790571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.791211 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.791791 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.791797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.792256 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.792875 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.792893 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.793709 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.793712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.794756 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.794786 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.795547 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.795553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.795752 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.796517 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.796529 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.799480 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.801468 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.801474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.801478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.801848 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.802343 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.802348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.802353 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.802489 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.803203 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.803205 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.803444 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.804722 (+ 1278us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.804836 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":281,"spinlock_wait_cycles":56832,"thread_start_us":610,"threads_started":1}
I20260927 18:47:12.805787 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.789520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.66:34419)
0927 18:47:12.790114 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.790163 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.791267 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.791794 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.791799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.792250 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:12.792892 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.792899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.794025 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.794029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.795045 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.795052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.795717 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.795721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.795862 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.796487 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:12.796508 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:12.799041 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:12.801373 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.801378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.801395 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.801848 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.802302 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:12.802308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:12.802311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:12.802494 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:12.803136 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:12.803141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:12.803444 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.804296 (+ 852us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.804692 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":258,"spinlock_wait_cycles":33024,"thread_start_us":157,"threads_started":1}
I20260927 18:47:12.807366 5068 heartbeater.cc:347] Connected to a master server at 127.29.159.124:44863
I20260927 18:47:12.807384 5070 heartbeater.cc:347] Connected to a master server at 127.29.159.125:40555
I20260927 18:47:12.807669 5068 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:12.807670 5070 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:12.808205 5068 heartbeater.cc:511] Master 127.29.159.124:44863 requested a full tablet report, sending...
I20260927 18:47:12.808208 5070 heartbeater.cc:511] Master 127.29.159.125:40555 requested a full tablet report, sending...
I20260927 18:47:12.809306 4752 ts_manager.cc:194] Registered new tserver with Master: d4346c5843c646e4bbf1b621a5d0c875 (127.29.159.66:43979)
I20260927 18:47:12.809357 4568 ts_manager.cc:194] Registered new tserver with Master: d4346c5843c646e4bbf1b621a5d0c875 (127.29.159.66:43979)
I20260927 18:47:12.810364 5069 heartbeater.cc:347] Connected to a master server at 127.29.159.126:40221
I20260927 18:47:12.810449 5069 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:12.810434 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:35793
--local_ip_for_outbound_sockets=127.29.159.67
--tserver_master_addrs=127.29.159.126:40221
--webserver_port=38977
--webserver_interface=127.29.159.67
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab
--principal=oryx/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.159.126:40221,127.29.159.125:40555
--tserver_master_addrs=127.29.159.126:40221,127.29.159.125:40555,127.29.159.124:44863 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:47:12.810632 5069 heartbeater.cc:511] Master 127.29.159.126:40221 requested a full tablet report, sending...
I20260927 18:47:12.811302 3968 ts_manager.cc:194] Re-registered known tserver with Master: d4346c5843c646e4bbf1b621a5d0c875 (127.29.159.66:43979)
I20260927 18:47:12.811975 3968 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:34419
I20260927 18:47:12.818567 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.814792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.29.159.124:44863)
0927 18:47:12.815009 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:12.815013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:12.815025 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:47:12.815063 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:12.815066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:12.815119 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:12.815212 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:12.815530 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.816022 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.817621 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.818020 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.818124 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.818217 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:12.818428 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 18:47:12.818481 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260927 18:47:12.818666 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:12.814582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:44863 (local address 127.0.0.1:53804)
0927 18:47:12.814760 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:12.814775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:12.814851 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:12.815253 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:12.815259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:12.815284 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:12.815400 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.815410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.816130 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.816137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.817498 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:12.817512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:12.818147 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:12.818153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:12.818276 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:12.818313 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:47:12.818400 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:47:12.818500 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 18:47:12.819276 4762 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 18:47:12.819661 4791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af467bdd4fc4d3f90cf79667730d4f3. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:12.819814 4791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: This master's current role is: LEARNER
W20260927 18:47:12.923870 5075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:12.924131 5075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:12.924182 5075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:12.928190 5075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:12.928243 5075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:12.928301 5075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:12.933601 5075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab
--principal=oryx/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:35793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=38977
--tserver_master_addrs=127.29.159.126:40221,127.29.159.125:40555,127.29.159.124:44863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:12.934616 5075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:12.935621 5075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:12.945528 5080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.945582 5081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:12.945582 5083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:12.946789 5075 server_base.cc:1034] running on GCE node
I20260927 18:47:12.947213 5075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:12.947901 5075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:12.949092 5075 hybrid_clock.cc:648] HybridClock initialized: now 1790534832949074 us; error 35 us; skew 500 ppm
Sep 27 18:47:12 dist-test-slave-8tg2 krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:12.952015 5075 init.cc:377] Logged in from keytab as oryx/127.29.159.67@KRBTEST.COM (short username oryx)
I20260927 18:47:12.952970 5075 webserver.cc:460] Webserver started at http://127.29.159.67:38977/ using document root <none> and password file <none>
I20260927 18:47:12.953548 5075 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:12.953655 5075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:12.957794 5075 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:47:12.960203 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.961493 5075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:12.961642 5075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "7fae20bb731f4ce2bea52eaa0418c7d1"
format_stamp: "Formatted at 2026-09-27 18:47:11 on dist-test-slave-8tg2"
server_key: "630d5b9cd47d6f3d5b3ca061485c95df"
server_key_iv: "ab827eb36ef1224ab18ef420e1ab2251"
server_key_version: "encryptionkey@0"
I20260927 18:47:12.962261 5075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:12.993250 5075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:12.996228 5075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:12.996430 5075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:12.997094 5075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:12.997944 5075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:12.998014 5075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:12.998095 5075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:12.998157 5075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:13.007496 5203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:35793 every 8 connection(s)
I20260927 18:47:13.007520 5075 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:35793
I20260927 18:47:13.008584 5075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:13.017478 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 5075
Sep 27 18:47:13 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
Sep 27 18:47:13 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.124@KRBTEST.COM
Sep 27 18:47:13 dist-test-slave-8tg2 krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534832, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.125@KRBTEST.COM
I20260927 18:47:13.027199 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.011020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:44863 (local address 127.29.159.67:41247)
0927 18:47:13.012080 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:13.012093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:13.012528 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:13.013906 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:13.013908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:13.014068 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:13.014645 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.014664 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.015787 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.015789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.017066 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.017077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.018231 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.018235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.018362 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.018862 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:13.018888 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:13.021535 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:13.023686 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:13.023692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:13.023720 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:13.024081 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:13.024418 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:13.024421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:13.024422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:13.024589 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:13.025044 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:13.025050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:13.025383 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 18:47:13.025601 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 18:47:13.025983 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":259,"spinlock_wait_cycles":3328,"thread_start_us":658,"threads_started":1}
I20260927 18:47:13.027716 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.010926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:40555 (local address 127.29.159.67:50973)
0927 18:47:13.011412 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:13.011490 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:13.012516 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:13.013294 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:13.013303 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:13.013895 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:13.014645 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.014662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.015692 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.015697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.017067 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.017077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.017722 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.017726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.017839 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.018888 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:13.018904 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:13.022304 (+ 3400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:13.024615 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:13.024618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:13.024620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:13.024879 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:13.026229 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:13.026233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:13.026235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:13.026328 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:13.027113 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:13.027119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:13.027222 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 18:47:13.027340 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 18:47:13.027507 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":315,"spinlock_wait_cycles":15104,"thread_start_us":150,"threads_started":1}
I20260927 18:47:13.028116 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.010595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:41247 (local address 127.29.159.124:44863)
0927 18:47:13.010732 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.010736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.012180 (+ 1444us) server_negotiation.cc:407] Connection header received
0927 18:47:13.013614 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.013621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.013692 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.013800 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:13.014818 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.015672 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.017533 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.017855 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.017941 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.021697 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:13.021701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:13.021703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:13.021733 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:13.023549 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.024215 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.024218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.024220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.024319 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.024695 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.024699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.024700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.024914 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:13.024994 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.027775 (+ 2781us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.027913 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:47:13.029178 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.010453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:50973 (local address 127.29.159.125:40555)
0927 18:47:13.010581 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.010586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.011594 (+ 1008us) server_negotiation.cc:407] Connection header received
0927 18:47:13.012770 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.012773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.012833 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.012890 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:13.014894 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.015545 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.017192 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.017515 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.017612 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.022646 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:13.022650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:13.022653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:13.022683 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:13.024493 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.026038 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.026042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.026044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.026126 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.026430 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.026433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.026435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.026963 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:13.027064 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.028856 (+ 1792us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.028994 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:47:13.029507 5204 heartbeater.cc:347] Connected to a master server at 127.29.159.124:44863
I20260927 18:47:13.029750 5204 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:13.030037 5206 heartbeater.cc:347] Connected to a master server at 127.29.159.125:40555
I20260927 18:47:13.030143 5206 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:13.030354 5206 heartbeater.cc:511] Master 127.29.159.125:40555 requested a full tablet report, sending...
I20260927 18:47:13.030634 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.020258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.29.159.125:40555)
0927 18:47:13.020538 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.020550 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.021549 (+ 999us) server_negotiation.cc:407] Connection header received
0927 18:47:13.021594 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.021597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.021658 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.021743 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:13.022785 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.023677 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.024826 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.025156 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.025244 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.026848 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:13.026852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:13.026854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:13.026877 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:13.028571 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.029030 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.029033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.029035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.029120 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.029440 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.029445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.029448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.029665 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:13.029789 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.030379 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.030499 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260927 18:47:13.031765 4567 ts_manager.cc:194] Registered new tserver with Master: 7fae20bb731f4ce2bea52eaa0418c7d1 (127.29.159.67:35793)
I20260927 18:47:13.030354 5204 heartbeater.cc:511] Master 127.29.159.124:44863 requested a full tablet report, sending...
I20260927 18:47:13.033118 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.010926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.67:46599)
0927 18:47:13.011943 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:13.011956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:13.012516 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:13.013294 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:13.013303 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:13.014362 (+ 1059us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:13.014645 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.014664 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.015828 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.015831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.016846 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.016855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.017450 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.017456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.017679 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.018426 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:13.018449 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:13.020635 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:13.029990 (+ 9355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:13.029995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:13.029997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:13.031151 (+ 1154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:13.031965 (+ 814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:13.031968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:13.031970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:13.032123 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:13.032631 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:13.032634 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:13.032720 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 18:47:13.032824 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:47:13.032950 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":196,"thread_start_us":154,"threads_started":1}
I20260927 18:47:13.033339 4752 ts_manager.cc:194] Registered new tserver with Master: 7fae20bb731f4ce2bea52eaa0418c7d1 (127.29.159.67:35793)
I20260927 18:47:13.034215 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.010697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:46599 (local address 127.29.159.126:40221)
0927 18:47:13.010841 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.010846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.012050 (+ 1204us) server_negotiation.cc:407] Connection header received
0927 18:47:13.012780 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.012784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.012833 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.012890 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:13.014824 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.015667 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.016985 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.017286 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.017389 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.020850 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:13.020855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:13.020858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:13.020889 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:13.025927 (+ 5038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.031267 (+ 5340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.031270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.031272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.031361 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.032228 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.032231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.032233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.032498 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:13.032580 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.033906 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.034019 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:47:13.034497 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.020689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.29.159.124:44863)
0927 18:47:13.022539 (+ 1850us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.022545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.022559 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:47:13.022598 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.022601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.022646 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.023682 (+ 1036us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:13.023796 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.024621 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.025852 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.026181 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.026642 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.027839 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:13.027843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:13.027845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:13.027870 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:13.029585 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.032887 (+ 3302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.032892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.032897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.032984 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.033311 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.033314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.033315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.033533 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:13.033666 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.034200 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.034332 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1765,"thread_start_us":89,"threads_started":1}
I20260927 18:47:13.034828 5205 heartbeater.cc:347] Connected to a master server at 127.29.159.126:40221
I20260927 18:47:13.034909 5205 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:13.035066 5205 heartbeater.cc:511] Master 127.29.159.126:40221 requested a full tablet report, sending...
I20260927 18:47:13.035738 3968 ts_manager.cc:194] Re-registered known tserver with Master: 7fae20bb731f4ce2bea52eaa0418c7d1 (127.29.159.67:35793)
I20260927 18:47:13.036284 3968 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:46599
I20260927 18:47:13.039098 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.021634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.29.159.126:40221)
0927 18:47:13.030262 (+ 8628us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.030266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.030280 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:47:13.030317 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.030320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.030363 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.030455 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:13.031260 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.032105 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.033269 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.033563 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.033677 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.036074 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:13.036078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:13.036080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:13.036112 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:13.037572 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.038025 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.038028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.038029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.038109 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.038361 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.038363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.038364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.038510 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:13.038604 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.038805 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.038942 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8515,"thread_start_us":81,"threads_started":1}
I20260927 18:47:13.041744 3968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:47:13.044893 3968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:47:13.047331 4718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: NON_VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: true } } }
I20260927 18:47:13.047498 4718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:13.050868 4718 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3: attempting to promote NON_VOTER 64bf2bba135d41f28b57fd32fc968054 to VOTER
I20260927 18:47:13.051304 4718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } }
I20260927 18:47:13.052455 4577 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Refusing update from remote peer 1af467bdd4fc4d3f90cf79667730d4f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 18:47:13.052569 4762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 LEARNER]: Refusing update from remote peer 1af467bdd4fc4d3f90cf79667730d4f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 18:47:13.052901 5222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 18:47:13.053292 5222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 18:47:13.055027 5222 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 64bf2bba135d41f28b57fd32fc968054 (127.29.159.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:13.055719 4577 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 64bf2bba135d41f28b57fd32fc968054 (127.29.159.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:13.055696 4762 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 64bf2bba135d41f28b57fd32fc968054 (127.29.159.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:13.056629 5222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:13.056733 5222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:13.056885 5222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:13.056998 5222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:13.056953 5224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:13.057076 5224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:13.057497 4791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:13.057632 4791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:13.057614 5232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:13.057801 5232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:13.066699 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.060865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:35793 (local address 127.0.0.1:39660)
0927 18:47:13.060999 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:13.061014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:13.061093 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:13.062957 (+ 1864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:13.062960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:13.062974 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:13.063055 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.063063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.064474 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.064476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.065174 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.065206 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.066272 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.066284 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.066400 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.066422 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:47:13.066486 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 18:47:13.066554 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":13}
I20260927 18:47:13.066823 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.061202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.29.159.67:35793)
0927 18:47:13.062024 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.062037 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.062062 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:47:13.062193 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.062201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.062394 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.063382 (+ 988us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:13.063600 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.064360 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.065340 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.066130 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.066374 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.066537 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.066586 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.066676 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":103,"threads_started":1}
I20260927 18:47:13.067158 5235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.061460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:43979 (local address 127.0.0.1:57780)
0927 18:47:13.061688 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:13.061700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:13.061812 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:13.062526 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:13.062532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:13.062551 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:13.062661 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.062667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.064284 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.064288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.065119 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.065128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.066273 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.066276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.066896 (+ 620us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.066922 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:47:13.066991 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:47:13.067062 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":18,"thread_start_us":90,"threads_started":1}
I20260927 18:47:13.067871 5233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.061207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:45531 (local address 127.0.0.1:51196)
0927 18:47:13.061660 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:13.061675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:13.061773 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:13.062724 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:13.062729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:13.062747 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:13.062839 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.062847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.065274 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.065277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.066151 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.066159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.067515 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.067518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.067607 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.067627 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:47:13.067706 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:47:13.067764 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":103,"threads_started":1}
I20260927 18:47:13.067884 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.061591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.29.159.66:43979)
0927 18:47:13.061966 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.061976 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.061995 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:47:13.062081 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.062087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.062373 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.062543 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:13.062774 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.064162 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.065265 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.066130 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.067414 (+ 1284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.067586 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.067636 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.067746 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
I20260927 18:47:13.067986 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.061949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.29.159.65:45531)
0927 18:47:13.062215 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.062228 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.062247 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:47:13.062313 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.062317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.062468 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.062585 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:13.062967 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.064154 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.066291 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.067179 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.067298 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.067468 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.067745 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.067838 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260927 18:47:13.070183 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 997dab9600624efa944eb2c0500ae858 (DEFAULT_TABLE table=test-table [id=a507d23bb4234f5dbd93600e1b64d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:13.070513 5002 tablet_service.cc:1467] Processing CreateTablet for tablet 997dab9600624efa944eb2c0500ae858 (DEFAULT_TABLE table=test-table [id=a507d23bb4234f5dbd93600e1b64d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:13.070590 4865 tablet_service.cc:1467] Processing CreateTablet for tablet 997dab9600624efa944eb2c0500ae858 (DEFAULT_TABLE table=test-table [id=a507d23bb4234f5dbd93600e1b64d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:13.071311 5138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997dab9600624efa944eb2c0500ae858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:13.071528 5002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997dab9600624efa944eb2c0500ae858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:13.071528 4865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997dab9600624efa944eb2c0500ae858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:13.079864 5238 tablet_bootstrap.cc:492] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: Bootstrap starting.
I20260927 18:47:13.080816 5240 tablet_bootstrap.cc:492] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878: Bootstrap starting.
I20260927 18:47:13.083124 5240 tablet_bootstrap.cc:654] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:13.084007 5240 log.cc:824] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:13.084702 5239 tablet_bootstrap.cc:492] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: Bootstrap starting.
I20260927 18:47:13.085856 5240 tablet_bootstrap.cc:492] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878: No bootstrap required, opened a new log
I20260927 18:47:13.086066 5240 ts_tablet_manager.cc:1397] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:47:13.086829 5239 tablet_bootstrap.cc:654] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:13.087610 5239 log.cc:824] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:13.089566 5238 tablet_bootstrap.cc:654] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:13.089397 5240 raft_consensus.cc:348] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } } peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } }
I20260927 18:47:13.089655 5240 raft_consensus.cc:374] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:13.089715 5240 raft_consensus.cc:729] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf85e20a1ec8466ca9bcd09b7df90878, State: Initialized, Role: FOLLOWER
I20260927 18:47:13.090167 5240 consensus_queue.cc:260] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } } peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } }
I20260927 18:47:13.090368 5238 log.cc:824] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:13.091115 5240 ts_tablet_manager.cc:1428] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:47:13.091519 4932 heartbeater.cc:503] Master 127.29.159.126:40221 was elected leader, sending a full tablet report...
I20260927 18:47:13.092036 5239 tablet_bootstrap.cc:492] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: No bootstrap required, opened a new log
I20260927 18:47:13.092201 5239 ts_tablet_manager.cc:1397] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 18:47:13.096414 5239 raft_consensus.cc:348] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } } peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } }
I20260927 18:47:13.096741 5239 raft_consensus.cc:374] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:13.096815 5239 raft_consensus.cc:729] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4346c5843c646e4bbf1b621a5d0c875, State: Initialized, Role: FOLLOWER
I20260927 18:47:13.097261 5239 consensus_queue.cc:260] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } } peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } }
I20260927 18:47:13.097509 5238 tablet_bootstrap.cc:492] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: No bootstrap required, opened a new log
I20260927 18:47:13.097755 5238 ts_tablet_manager.cc:1397] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260927 18:47:13.098335 5239 ts_tablet_manager.cc:1428] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 18:47:13.099483 5069 heartbeater.cc:503] Master 127.29.159.126:40221 was elected leader, sending a full tablet report...
I20260927 18:47:13.102994 5238 raft_consensus.cc:348] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } } peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } }
I20260927 18:47:13.103243 5238 raft_consensus.cc:374] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:13.103307 5238 raft_consensus.cc:729] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fae20bb731f4ce2bea52eaa0418c7d1, State: Initialized, Role: FOLLOWER
I20260927 18:47:13.103740 5238 consensus_queue.cc:260] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } } peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } }
I20260927 18:47:13.104387 5205 heartbeater.cc:503] Master 127.29.159.126:40221 was elected leader, sending a full tablet report...
I20260927 18:47:13.104692 5238 ts_tablet_manager.cc:1428] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 18:47:13.234107 5243 raft_consensus.cc:482] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:13.234364 5243 raft_consensus.cc:504] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } } peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } }
I20260927 18:47:13.235584 5243 leader_election.cc:288] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fae20bb731f4ce2bea52eaa0418c7d1 (127.29.159.67:35793), d4346c5843c646e4bbf1b621a5d0c875 (127.29.159.66:43979)
I20260927 18:47:13.239650 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.235844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:43979 (local address 127.29.159.65:51335)
0927 18:47:13.236169 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:13.236188 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:13.236284 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:13.236563 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:13.236565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:13.236582 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:13.236710 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.236717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.237503 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.237506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.238366 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.238375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.239077 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.239081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.239249 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.239268 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:47:13.239371 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 18:47:13.239474 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":82,"threads_started":1}
I20260927 18:47:13.239675 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.236011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:51335 (local address 127.29.159.66:43979)
0927 18:47:13.236182 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.236187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.236243 (+ 56us) server_negotiation.cc:407] Connection header received
0927 18:47:13.236404 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.236408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.236464 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.236539 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:13.236828 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.237390 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.238513 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.238952 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.239086 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.239143 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.239414 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.239549 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 18:47:13.239792 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.235942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:53473 (local address 127.29.159.67:35793)
0927 18:47:13.236076 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.236081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.236367 (+ 286us) server_negotiation.cc:407] Connection header received
0927 18:47:13.236508 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.236532 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.236586 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.236690 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:13.236974 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.237509 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.238665 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.239223 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.239324 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.239400 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.239605 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.239674 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:47:13.239790 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.235845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:35793 (local address 127.29.159.65:53473)
0927 18:47:13.236244 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:13.236259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:13.236397 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:13.236717 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:13.236719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:13.236734 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:13.236863 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.236869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.237645 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.237648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.238504 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:13.238513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.239345 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.239351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:13.239465 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.239484 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:47:13.239554 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 18:47:13.239614 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":31,"thread_start_us":77,"threads_started":1}
I20260927 18:47:13.240489 5158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997dab9600624efa944eb2c0500ae858" candidate_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" is_pre_election: true
I20260927 18:47:13.240492 5022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997dab9600624efa944eb2c0500ae858" candidate_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4346c5843c646e4bbf1b621a5d0c875" is_pre_election: true
I20260927 18:47:13.240795 5022 raft_consensus.cc:2454] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf85e20a1ec8466ca9bcd09b7df90878 in term 0.
I20260927 18:47:13.240808 5158 raft_consensus.cc:2454] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf85e20a1ec8466ca9bcd09b7df90878 in term 0.
I20260927 18:47:13.241261 4820 leader_election.cc:302] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf85e20a1ec8466ca9bcd09b7df90878, d4346c5843c646e4bbf1b621a5d0c875; no voters:
I20260927 18:47:13.241544 5243 raft_consensus.cc:2781] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:13.241626 5243 raft_consensus.cc:482] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:13.241669 5243 raft_consensus.cc:3037] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:13.242697 5243 raft_consensus.cc:504] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } } peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } }
I20260927 18:47:13.243022 5243 leader_election.cc:288] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [CANDIDATE]: Term 1 election: Requested vote from peers 7fae20bb731f4ce2bea52eaa0418c7d1 (127.29.159.67:35793), d4346c5843c646e4bbf1b621a5d0c875 (127.29.159.66:43979)
I20260927 18:47:13.243407 5158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997dab9600624efa944eb2c0500ae858" candidate_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1"
I20260927 18:47:13.243407 5022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997dab9600624efa944eb2c0500ae858" candidate_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4346c5843c646e4bbf1b621a5d0c875"
I20260927 18:47:13.243566 5158 raft_consensus.cc:3037] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:13.243566 5022 raft_consensus.cc:3037] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:13.244931 5158 raft_consensus.cc:2454] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf85e20a1ec8466ca9bcd09b7df90878 in term 1.
I20260927 18:47:13.244931 5022 raft_consensus.cc:2454] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf85e20a1ec8466ca9bcd09b7df90878 in term 1.
I20260927 18:47:13.245244 4818 leader_election.cc:302] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7fae20bb731f4ce2bea52eaa0418c7d1, cf85e20a1ec8466ca9bcd09b7df90878, d4346c5843c646e4bbf1b621a5d0c875; no voters:
I20260927 18:47:13.245453 5243 raft_consensus.cc:2781] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:13.245666 5243 raft_consensus.cc:686] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 1 LEADER]: Becoming Leader. State: Replica: cf85e20a1ec8466ca9bcd09b7df90878, State: Running, Role: LEADER
I20260927 18:47:13.246029 5243 consensus_queue.cc:237] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } } peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } }
I20260927 18:47:13.249734 3968 catalog_manager.cc:5495] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf85e20a1ec8466ca9bcd09b7df90878 (127.29.159.65). New cstate: current_term: 1 leader_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } health_report { overall_health: UNKNOWN } } }
W20260927 18:47:13.260010 5207 tablet.cc:2369] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:47:13.289422 5071 tablet.cc:2369] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:13.302878 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.299471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51208 (local address 127.29.159.65:45531)
0927 18:47:13.299650 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.299655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.299671 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:47:13.299785 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.299787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.299860 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.299983 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:47:13.300224 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.300742 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.301556 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.301789 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.301871 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.302128 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:47:13.302573 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:47:13.302690 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.302719 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.302770 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:47:13.308827 5022 raft_consensus.cc:1261] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875 [term 1 FOLLOWER]: Refusing update from remote peer cf85e20a1ec8466ca9bcd09b7df90878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:13.308833 5158 raft_consensus.cc:1261] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 1 FOLLOWER]: Refusing update from remote peer cf85e20a1ec8466ca9bcd09b7df90878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:13.309558 5243 consensus_queue.cc:1035] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:13.309736 5249 consensus_queue.cc:1035] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:13.317783 5252 mvcc.cc:205] Tried to move back new op lower bound from 7334030677225660416 to 7334030676978663424. Current Snapshot: MvccSnapshot[applied={T|T < 7334030677225660416}]
I20260927 18:47:13.320990 5254 mvcc.cc:205] Tried to move back new op lower bound from 7334030677225660416 to 7334030676978663424. Current Snapshot: MvccSnapshot[applied={T|T < 7334030677225660416}]
I20260927 18:47:13.322770 5253 mvcc.cc:205] Tried to move back new op lower bound from 7334030677225660416 to 7334030676978663424. Current Snapshot: MvccSnapshot[applied={T|T < 7334030677225660416}]
I20260927 18:47:13.330792 3968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:13.330991 3968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:13.335685 3968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1af467bdd4fc4d3f90cf79667730d4f3: Sending DeleteTablet for 3 replicas of tablet 997dab9600624efa944eb2c0500ae858
I20260927 18:47:13.336550 5002 tablet_service.cc:1514] Processing DeleteTablet for tablet 997dab9600624efa944eb2c0500ae858 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:13 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:57780
I20260927 18:47:13.336896 5138 tablet_service.cc:1514] Processing DeleteTablet for tablet 997dab9600624efa944eb2c0500ae858 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:13 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:39660
I20260927 18:47:13.336946 4865 tablet_service.cc:1514] Processing DeleteTablet for tablet 997dab9600624efa944eb2c0500ae858 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:13 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:51196
I20260927 18:47:13.337697 5262 tablet_replica.cc:330] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878: stopping tablet replica
I20260927 18:47:13.338059 5262 raft_consensus.cc:2229] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:47:13.338675 5262 raft_consensus.cc:2258] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:13.338920 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 3936
I20260927 18:47:13.339010 5263 tablet_replica.cc:330] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: stopping tablet replica
I20260927 18:47:13.339908 5263 raft_consensus.cc:2229] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:13.340221 5263 raft_consensus.cc:2258] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:13.341281 5262 ts_tablet_manager.cc:1905] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:13.343616 5262 ts_tablet_manager.cc:1918] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:13.343722 5262 log.cc:1196] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-0/wal/wals/997dab9600624efa944eb2c0500ae858
I20260927 18:47:13.344020 5262 ts_tablet_manager.cc:1939] T 997dab9600624efa944eb2c0500ae858 P cf85e20a1ec8466ca9bcd09b7df90878: Deleting consensus metadata
I20260927 18:47:13.346025 5263 ts_tablet_manager.cc:1905] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 18:47:13.346654 4818 connection.cc:535] server connection from 127.0.0.1:51196 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:47:13.348707 5263 ts_tablet_manager.cc:1918] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:13.348811 5263 log.cc:1196] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/997dab9600624efa944eb2c0500ae858
I20260927 18:47:13.349169 5263 ts_tablet_manager.cc:1939] T 997dab9600624efa944eb2c0500ae858 P 7fae20bb731f4ce2bea52eaa0418c7d1: Deleting consensus metadata
W20260927 18:47:13.350281 5093 connection.cc:441] server connection from 127.0.0.1:39660 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39660 (request call id 1) could send its response
I20260927 18:47:13.351820 5261 tablet_replica.cc:330] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: stopping tablet replica
I20260927 18:47:13.352154 5261 raft_consensus.cc:2229] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:13.352468 5261 raft_consensus.cc:2258] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:13.354686 5261 ts_tablet_manager.cc:1905] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:13.357553 5261 ts_tablet_manager.cc:1918] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:13.357654 5261 log.cc:1196] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/997dab9600624efa944eb2c0500ae858
I20260927 18:47:13.357965 5261 ts_tablet_manager.cc:1939] T 997dab9600624efa944eb2c0500ae858 P d4346c5843c646e4bbf1b621a5d0c875: Deleting consensus metadata
I20260927 18:47:13.358613 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.338583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.29.159.125:40555)
0927 18:47:13.338696 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.338700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.348285 (+ 9585us) server_negotiation.cc:407] Connection header received
0927 18:47:13.348473 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.348476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.348528 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.348579 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:13.349586 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.350414 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.351659 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.352058 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.352169 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.354644 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:13.354648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:13.354650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:13.354678 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:13.356572 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.357079 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.357083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.357085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.357170 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.357451 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.357453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.357455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.357671 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:13.357782 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.358319 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.358446 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 18:47:13.359289 4955 connection.cc:441] server connection from 127.0.0.1:57780 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57780 (request call id 1) could send its response
I20260927 18:47:13.365365 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.350558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.29.159.125:40555)
0927 18:47:13.350665 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.350670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.352265 (+ 1595us) server_negotiation.cc:407] Connection header received
0927 18:47:13.352302 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.352307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.352348 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.352418 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:13.353878 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.354682 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.358698 (+ 4016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.359029 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.359107 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.361602 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:13.361614 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:13.361620 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:13.361651 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:13.363312 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.363898 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.363904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.363907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.363989 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.364344 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.364350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.364352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.364568 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:13.364693 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.365059 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.365190 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 18:47:13.367108 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.351231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.29.159.124:44863)
0927 18:47:13.351374 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.351378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.352317 (+ 939us) server_negotiation.cc:407] Connection header received
0927 18:47:13.353363 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.353368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.353432 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.353502 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:13.356716 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.357513 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.359639 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.359940 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.360024 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.362982 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:13.362986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:13.362992 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:13.363021 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:13.364845 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.365589 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.365595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.365598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.365691 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.365986 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.365990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.365991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.366225 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:13.366355 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.366843 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.366953 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:47:13.369196 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:13.338563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.29.159.124:44863)
0927 18:47:13.338686 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:13.338693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:13.360295 (+ 21602us) server_negotiation.cc:407] Connection header received
0927 18:47:13.360471 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:13.360474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:13.360530 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:13.360615 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:13.361288 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.362062 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.363190 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:13.363677 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:13.363815 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:13.365729 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:13.365733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:13.365736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:13.365762 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:13.367649 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.368072 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.368076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.368078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.368157 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:13.368420 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:13.368423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:13.368425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:13.368618 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:13.368738 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:13.368932 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 18:47:13.369047 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:47:14.322650 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.322083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.67:53037)
0927 18:47:14.322387 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.322494 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":79,"threads_started":1}
W20260927 18:47:14.322996 5205 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:40221 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
I20260927 18:47:14.324457 5273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.324010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.65:44303)
0927 18:47:14.324301 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.324376 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":137,"threads_started":1}
W20260927 18:47:14.324796 4932 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:40221 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
I20260927 18:47:14.325301 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.324918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.66:35817)
0927 18:47:14.325160 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.325225 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":65,"threads_started":1}
W20260927 18:47:14.325629 5069 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:40221 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
I20260927 18:47:14.835733 5275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1af467bdd4fc4d3f90cf79667730d4f3)
I20260927 18:47:14.836020 5275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } }
I20260927 18:47:14.836237 5276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1af467bdd4fc4d3f90cf79667730d4f3)
I20260927 18:47:14.836452 5276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } }
I20260927 18:47:14.837284 5275 leader_election.cc:288] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221), 64bf2bba135d41f28b57fd32fc968054 (127.29.159.124:44863)
I20260927 18:47:14.837354 5276 leader_election.cc:288] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221), 59f72f81f60a469c8b9fdae0433cc518 (127.29.159.125:40555)
I20260927 18:47:14.838064 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.837611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36438)
0927 18:47:14.837869 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.837955 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":54,"threads_started":1}
W20260927 18:47:14.838253 4739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
I20260927 18:47:14.838323 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.837563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36428)
0927 18:47:14.838133 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.838214 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":80,"threads_started":1}
W20260927 18:47:14.838553 4555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
I20260927 18:47:14.839190 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.838948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36448)
0927 18:47:14.839077 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.839126 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 18:47:14.839337 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.839099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36456)
0927 18:47:14.839219 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.839276 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:47:14.839329 4739 leader_election.cc:334] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221): Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
W20260927 18:47:14.839473 4555 leader_election.cc:334] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221): Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
I20260927 18:47:14.842299 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.837900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.29.159.124:44863)
0927 18:47:14.838139 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:14.838146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:14.838162 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:47:14.838214 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:14.838218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:14.838278 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:14.838380 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:14.838780 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:14.839665 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:14.841248 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:14.841705 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:14.841840 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:14.841894 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:14.842076 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 18:47:14.842167 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":56,"threads_started":1}
I20260927 18:47:14.842312 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.837526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:44863 (local address 127.0.0.1:53850)
0927 18:47:14.837869 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.837885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:14.837989 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:14.838501 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:14.838504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:14.838524 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:14.838648 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:14.838656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:14.839797 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:14.839801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:14.841128 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:14.841136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:14.841828 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:14.841832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:14.841932 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:14.841965 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:47:14.842060 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:47:14.842175 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":23,"thread_start_us":74,"threads_started":1}
I20260927 18:47:14.842864 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59f72f81f60a469c8b9fdae0433cc518" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "64bf2bba135d41f28b57fd32fc968054" is_pre_election: true
I20260927 18:47:14.843006 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.837617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:40555 (local address 127.0.0.1:39554)
0927 18:47:14.838319 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.838372 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:14.839227 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:14.839776 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:14.839778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:14.839792 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:14.839855 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:14.839859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:14.840908 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:14.840910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:14.841846 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:14.841855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:14.842604 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:14.842608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:14.842716 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:14.842751 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:47:14.842832 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:47:14.842889 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":51,"threads_started":1}
I20260927 18:47:14.843050 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.837879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.29.159.125:40555)
0927 18:47:14.838610 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:14.838615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:14.839187 (+ 572us) server_negotiation.cc:407] Connection header received
0927 18:47:14.839528 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:14.839568 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:14.839616 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:14.839674 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:14.839950 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:14.840796 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:14.841974 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:14.842488 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:14.842600 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:14.842645 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:14.842856 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 18:47:14.842937 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":668,"thread_start_us":59,"threads_started":1}
I20260927 18:47:14.843087 4762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f72f81f60a469c8b9fdae0433cc518 in term 1.
I20260927 18:47:14.843461 4554 leader_election.cc:302] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f72f81f60a469c8b9fdae0433cc518, 64bf2bba135d41f28b57fd32fc968054; no voters: 1af467bdd4fc4d3f90cf79667730d4f3
I20260927 18:47:14.843523 4577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64bf2bba135d41f28b57fd32fc968054" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59f72f81f60a469c8b9fdae0433cc518" is_pre_election: true
I20260927 18:47:14.843716 5275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 18:47:14.843804 4577 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64bf2bba135d41f28b57fd32fc968054 in term 1.
I20260927 18:47:14.843861 5275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1af467bdd4fc4d3f90cf79667730d4f3)
I20260927 18:47:14.843917 5275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:47:14.844182 4736 leader_election.cc:302] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f72f81f60a469c8b9fdae0433cc518, 64bf2bba135d41f28b57fd32fc968054; no voters: 1af467bdd4fc4d3f90cf79667730d4f3
I20260927 18:47:14.844405 5276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 18:47:14.844476 5276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1af467bdd4fc4d3f90cf79667730d4f3)
I20260927 18:47:14.844533 5276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:47:14.845177 5275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } }
I20260927 18:47:14.845513 5276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } }
I20260927 18:47:14.845638 5275 leader_election.cc:288] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 2 election: Requested vote from peers 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221), 64bf2bba135d41f28b57fd32fc968054 (127.29.159.124:44863)
I20260927 18:47:14.845903 5276 leader_election.cc:288] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [CANDIDATE]: Term 2 election: Requested vote from peers 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221), 59f72f81f60a469c8b9fdae0433cc518 (127.29.159.125:40555)
I20260927 18:47:14.845990 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59f72f81f60a469c8b9fdae0433cc518" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "64bf2bba135d41f28b57fd32fc968054"
I20260927 18:47:14.846024 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.845702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36462)
0927 18:47:14.845889 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.845953 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 18:47:14.846130 4762 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f72f81f60a469c8b9fdae0433cc518 in current term 2: Already voted for candidate 64bf2bba135d41f28b57fd32fc968054 in this term.
I20260927 18:47:14.846315 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.846039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36466)
0927 18:47:14.846190 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.846253 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 18:47:14.846284 4577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64bf2bba135d41f28b57fd32fc968054" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59f72f81f60a469c8b9fdae0433cc518"
I20260927 18:47:14.846436 4577 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 64bf2bba135d41f28b57fd32fc968054 in current term 2: Already voted for candidate 59f72f81f60a469c8b9fdae0433cc518 in this term.
I20260927 18:47:14.846767 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.846512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36472)
0927 18:47:14.846649 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.846704 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 18:47:14.846868 4555 leader_election.cc:334] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221): Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
I20260927 18:47:14.846933 4555 leader_election.cc:302] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f72f81f60a469c8b9fdae0433cc518; no voters: 1af467bdd4fc4d3f90cf79667730d4f3, 64bf2bba135d41f28b57fd32fc968054
I20260927 18:47:14.847069 5275 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 18:47:14.847139 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:14.846828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36488)
0927 18:47:14.846990 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:14.847052 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 18:47:14.847237 4739 leader_election.cc:334] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221): Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
I20260927 18:47:14.847317 4739 leader_election.cc:302] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64bf2bba135d41f28b57fd32fc968054; no voters: 1af467bdd4fc4d3f90cf79667730d4f3, 59f72f81f60a469c8b9fdae0433cc518
I20260927 18:47:14.847499 5276 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 18:47:15.324021 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:15.323565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.67:36425)
0927 18:47:15.323862 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:15.323934 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
I20260927 18:47:15.325714 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:15.325358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.65:53237)
0927 18:47:15.325572 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:15.325636 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":58,"threads_started":1}
I20260927 18:47:15.326552 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:15.326124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.66:36293)
0927 18:47:15.326385 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:15.326449 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":67,"threads_started":1}
I20260927 18:47:16.325196 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.324692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.67:34293)
0927 18:47:16.325023 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.325096 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":81,"threads_started":1}
I20260927 18:47:16.326754 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.326369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.65:50735)
0927 18:47:16.326618 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.326675 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":66,"threads_started":1}
I20260927 18:47:16.327684 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.327254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.66:35425)
0927 18:47:16.327499 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.327549 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260927 18:47:16.452540 5291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:16.452682 5291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } }
I20260927 18:47:16.453280 5291 leader_election.cc:288] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221), 64bf2bba135d41f28b57fd32fc968054 (127.29.159.124:44863)
I20260927 18:47:16.453773 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59f72f81f60a469c8b9fdae0433cc518" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "64bf2bba135d41f28b57fd32fc968054" is_pre_election: true
I20260927 18:47:16.453817 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.453421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36704)
0927 18:47:16.453659 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.453730 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260927 18:47:16.453902 4762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f72f81f60a469c8b9fdae0433cc518 in term 2.
I20260927 18:47:16.454247 4554 leader_election.cc:302] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f72f81f60a469c8b9fdae0433cc518, 64bf2bba135d41f28b57fd32fc968054; no voters:
I20260927 18:47:16.454428 5291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 18:47:16.454500 5291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:16.454566 5291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:47:16.454826 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.454571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36708)
0927 18:47:16.454734 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.454776 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 18:47:16.454936 4555 leader_election.cc:334] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221): Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
I20260927 18:47:16.455762 5291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } }
I20260927 18:47:16.456079 5291 leader_election.cc:288] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 3 election: Requested vote from peers 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221), 64bf2bba135d41f28b57fd32fc968054 (127.29.159.124:44863)
I20260927 18:47:16.456492 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.456285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36720)
0927 18:47:16.456394 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.456443 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 18:47:16.456499 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59f72f81f60a469c8b9fdae0433cc518" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "64bf2bba135d41f28b57fd32fc968054"
I20260927 18:47:16.456643 4762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:47:16.457106 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.456906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36726)
0927 18:47:16.457011 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.457055 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 18:47:16.457244 4555 leader_election.cc:334] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221): Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
I20260927 18:47:16.457726 4762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f72f81f60a469c8b9fdae0433cc518 in term 3.
I20260927 18:47:16.458328 4554 leader_election.cc:302] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f72f81f60a469c8b9fdae0433cc518, 64bf2bba135d41f28b57fd32fc968054; no voters: 1af467bdd4fc4d3f90cf79667730d4f3
I20260927 18:47:16.458519 5291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:47:16.458729 5291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [term 3 LEADER]: Becoming Leader. State: Replica: 59f72f81f60a469c8b9fdae0433cc518, State: Running, Role: LEADER
I20260927 18:47:16.458997 5291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } }
I20260927 18:47:16.460132 5294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59f72f81f60a469c8b9fdae0433cc518. Latest consensus state: current_term: 3 leader_uuid: "59f72f81f60a469c8b9fdae0433cc518" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:16.460269 5294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:16.460556 5296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:16.462706 5296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:16.463110 5296 catalog_manager.cc:1259] Loaded cluster ID: 3cd05ce26e8f4e80ac24854feb0b86a2
I20260927 18:47:16.463156 5296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:16.463786 5296 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:16.464178 5296 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: Loaded TSK: 0
I20260927 18:47:16.464870 5296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:47:16.491322 4567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:47:16.494693 4567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:47:16.497711 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.497400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36754)
0927 18:47:16.497538 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.497598 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 18:47:16.497712 4762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [term 3 FOLLOWER]: Refusing update from remote peer 59f72f81f60a469c8b9fdae0433cc518: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 18:47:16.498186 5291 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 18:47:16.498477 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.498241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.0.0.1:36762)
0927 18:47:16.498333 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.498415 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 18:47:16.498603 4555 consensus_peers.cc:487] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 -> Peer 1af467bdd4fc4d3f90cf79667730d4f3 (127.29.159.126:40221): Couldn't send request to peer 1af467bdd4fc4d3f90cf79667730d4f3. Status: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:47:16.499816 5298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59f72f81f60a469c8b9fdae0433cc518. Latest consensus state: current_term: 3 leader_uuid: "59f72f81f60a469c8b9fdae0433cc518" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:16.499960 5298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:16.500936 5291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "59f72f81f60a469c8b9fdae0433cc518" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:16.501050 5291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:16.501166 5298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "59f72f81f60a469c8b9fdae0433cc518" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1af467bdd4fc4d3f90cf79667730d4f3" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40221 } } peers { permanent_uuid: "59f72f81f60a469c8b9fdae0433cc518" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 40555 } attrs { promote: false } } peers { permanent_uuid: "64bf2bba135d41f28b57fd32fc968054" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 44863 } attrs { promote: false } } }
I20260927 18:47:16.501328 5298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64bf2bba135d41f28b57fd32fc968054 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:47:16.513291 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.508910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:43979 (local address 127.0.0.1:57786)
0927 18:47:16.509086 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.509097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:16.509167 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:16.509594 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:16.509599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:16.509621 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:16.509729 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:16.509735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.510811 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.510817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:16.511987 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:16.511997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.512872 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.512876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:16.512963 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:16.512984 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:47:16.513055 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:47:16.513131 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":1}
I20260927 18:47:16.513938 5304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.509074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.29.159.66:43979)
0927 18:47:16.509325 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:16.509329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:16.509349 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:47:16.509411 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:16.509416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:16.509465 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:16.509849 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:16.509961 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.510677 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.512124 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.512755 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.512875 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:16.512941 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:16.513731 (+ 790us) server_negotiation.cc:299] Negotiation successful
0927 18:47:16.513826 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260927 18:47:16.513964 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.509388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:35793 (local address 127.0.0.1:39662)
0927 18:47:16.509624 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.509639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:16.509722 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:16.510043 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:16.510045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:16.510059 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:16.510321 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:16.510328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.511377 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.511380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:16.512513 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:16.512522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.513560 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.513564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:16.513663 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:16.513694 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:47:16.513750 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 18:47:16.513828 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260927 18:47:16.514318 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.509701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:45531 (local address 127.0.0.1:51216)
0927 18:47:16.510502 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.510513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:16.510584 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:16.510913 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:16.510916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:16.510932 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:16.511016 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:16.511021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.511968 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.511973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:16.512889 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:16.512898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.513925 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.513928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:16.514019 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:16.514047 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 18:47:16.514122 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:47:16.514197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260927 18:47:16.514472 5002 tablet_service.cc:1467] Processing CreateTablet for tablet aa069db4d4ca43ef9ae927bac4dc36a3 (DEFAULT_TABLE table=test-table [id=a1e4a17ccf38441fa2a3ac7415e8820f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:16.514653 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.509517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.29.159.67:35793)
0927 18:47:16.509748 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:16.509752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:16.509765 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:47:16.509817 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:16.509823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:16.509870 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:16.509946 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:16.510421 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.511239 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.512642 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.513410 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.514234 (+ 824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:16.514302 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:16.514341 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 18:47:16.514528 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260927 18:47:16.514689 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.510099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.29.159.65:45531)
0927 18:47:16.510336 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:16.510340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:16.510566 (+ 226us) server_negotiation.cc:407] Connection header received
0927 18:47:16.510660 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:16.510663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:16.510710 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:16.510810 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:16.511224 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.511851 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.513028 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.513710 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.513812 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:16.513870 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:16.514475 (+ 605us) server_negotiation.cc:299] Negotiation successful
0927 18:47:16.514554 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
I20260927 18:47:16.514834 5002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa069db4d4ca43ef9ae927bac4dc36a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:16.515218 5138 tablet_service.cc:1467] Processing CreateTablet for tablet aa069db4d4ca43ef9ae927bac4dc36a3 (DEFAULT_TABLE table=test-table [id=a1e4a17ccf38441fa2a3ac7415e8820f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:16.515496 5138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa069db4d4ca43ef9ae927bac4dc36a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:16.515692 4865 tablet_service.cc:1467] Processing CreateTablet for tablet aa069db4d4ca43ef9ae927bac4dc36a3 (DEFAULT_TABLE table=test-table [id=a1e4a17ccf38441fa2a3ac7415e8820f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:16.515951 4865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa069db4d4ca43ef9ae927bac4dc36a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:16.517455 5309 tablet_bootstrap.cc:492] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875: Bootstrap starting.
I20260927 18:47:16.518286 5310 tablet_bootstrap.cc:492] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1: Bootstrap starting.
I20260927 18:47:16.518512 5309 tablet_bootstrap.cc:654] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:16.519220 5310 tablet_bootstrap.cc:654] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:16.519665 5311 tablet_bootstrap.cc:492] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878: Bootstrap starting.
I20260927 18:47:16.519887 5309 tablet_bootstrap.cc:492] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875: No bootstrap required, opened a new log
I20260927 18:47:16.519958 5309 ts_tablet_manager.cc:1397] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:16.520352 5309 raft_consensus.cc:348] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } } peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } }
I20260927 18:47:16.520444 5309 raft_consensus.cc:374] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:16.520465 5309 raft_consensus.cc:729] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4346c5843c646e4bbf1b621a5d0c875, State: Initialized, Role: FOLLOWER
I20260927 18:47:16.520578 5311 tablet_bootstrap.cc:654] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:16.520599 5309 consensus_queue.cc:260] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } } peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } }
I20260927 18:47:16.521019 5309 ts_tablet_manager.cc:1428] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:16.521612 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.521318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.66:51319)
0927 18:47:16.521482 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.521537 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 18:47:16.522826 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.522284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.66:56749)
0927 18:47:16.522715 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.522771 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362}
I20260927 18:47:16.523550 5070 heartbeater.cc:503] Master 127.29.159.125:40555 was elected leader, sending a full tablet report...
I20260927 18:47:16.523597 5310 tablet_bootstrap.cc:492] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1: No bootstrap required, opened a new log
I20260927 18:47:16.523676 5310 ts_tablet_manager.cc:1397] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 18:47:16.524139 5310 raft_consensus.cc:348] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } } peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } }
I20260927 18:47:16.524252 5311 tablet_bootstrap.cc:492] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878: No bootstrap required, opened a new log
I20260927 18:47:16.524266 5310 raft_consensus.cc:374] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:16.524302 5310 raft_consensus.cc:729] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fae20bb731f4ce2bea52eaa0418c7d1, State: Initialized, Role: FOLLOWER
I20260927 18:47:16.524350 5311 ts_tablet_manager.cc:1397] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 18:47:16.524449 5310 consensus_queue.cc:260] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } } peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } }
I20260927 18:47:16.524812 5310 ts_tablet_manager.cc:1428] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:16.524803 5311 raft_consensus.cc:348] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } } peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } }
I20260927 18:47:16.524916 5311 raft_consensus.cc:374] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:16.524968 5311 raft_consensus.cc:729] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf85e20a1ec8466ca9bcd09b7df90878, State: Initialized, Role: FOLLOWER
I20260927 18:47:16.525092 5311 consensus_queue.cc:260] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } } peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } }
I20260927 18:47:16.525446 5311 ts_tablet_manager.cc:1428] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:16.525561 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.525300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.67:37923)
0927 18:47:16.525440 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.525494 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 18:47:16.526374 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.526038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.65:47087)
0927 18:47:16.526246 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.526302 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 18:47:16.527781 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.527412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.67:50781)
0927 18:47:16.527660 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.527708 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260927 18:47:16.528115 4933 heartbeater.cc:503] Master 127.29.159.125:40555 was elected leader, sending a full tablet report...
I20260927 18:47:16.528769 5206 heartbeater.cc:503] Master 127.29.159.125:40555 was elected leader, sending a full tablet report...
I20260927 18:47:16.528798 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.528519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.65:43205)
0927 18:47:16.528653 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.528728 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 18:47:16.577234 4934 tablet.cc:2369] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:16.744447 5316 raft_consensus.cc:482] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:16.744585 5316 raft_consensus.cc:504] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } } peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } }
I20260927 18:47:16.745682 5316 leader_election.cc:288] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4346c5843c646e4bbf1b621a5d0c875 (127.29.159.66:43979), cf85e20a1ec8466ca9bcd09b7df90878 (127.29.159.65:45531)
I20260927 18:47:16.749567 5304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.745989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:39851 (local address 127.29.159.66:43979)
0927 18:47:16.746138 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:16.746145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:16.746169 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:47:16.746229 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:16.746232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:16.746301 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:16.746371 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:16.746699 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.747308 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.748556 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.749019 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.749143 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:16.749202 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:16.749389 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 18:47:16.749460 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:47:16.749565 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.745866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:43979 (local address 127.29.159.67:39851)
0927 18:47:16.745990 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.746007 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:16.746116 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:16.746407 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:16.746411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:16.746439 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:16.746583 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:16.746590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.747450 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.747454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:16.748441 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:16.748450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.749136 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.749140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:16.749256 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:16.749274 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:47:16.749359 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:47:16.749429 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":54}
I20260927 18:47:16.749763 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.745995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:45531 (local address 127.29.159.67:52009)
0927 18:47:16.746399 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.746412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:16.746498 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:16.746764 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:16.746767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:16.746785 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:16.746868 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:16.746874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.747648 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.747655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:16.748451 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:16.748460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.749182 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.749186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:16.749337 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:16.749358 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:47:16.749475 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 18:47:16.749551 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":137,"threads_started":1}
I20260927 18:47:16.749823 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.746164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:52009 (local address 127.29.159.65:45531)
0927 18:47:16.746302 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:16.746307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:16.746495 (+ 188us) server_negotiation.cc:407] Connection header received
0927 18:47:16.746589 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:16.746592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:16.746667 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:16.746749 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:16.746976 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.747537 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.748580 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.749019 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.749149 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:16.749202 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:16.749505 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 18:47:16.749579 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:47:16.750094 5021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa069db4d4ca43ef9ae927bac4dc36a3" candidate_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4346c5843c646e4bbf1b621a5d0c875" is_pre_election: true
I20260927 18:47:16.750259 5021 raft_consensus.cc:2454] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fae20bb731f4ce2bea52eaa0418c7d1 in term 0.
I20260927 18:47:16.750341 4885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa069db4d4ca43ef9ae927bac4dc36a3" candidate_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" is_pre_election: true
I20260927 18:47:16.750564 4885 raft_consensus.cc:2454] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fae20bb731f4ce2bea52eaa0418c7d1 in term 0.
I20260927 18:47:16.750649 5093 leader_election.cc:302] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fae20bb731f4ce2bea52eaa0418c7d1, d4346c5843c646e4bbf1b621a5d0c875; no voters:
I20260927 18:47:16.750895 5316 raft_consensus.cc:2781] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:16.750952 5316 raft_consensus.cc:482] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:16.751078 5316 raft_consensus.cc:3037] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:16.752275 5316 raft_consensus.cc:504] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } } peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } }
I20260927 18:47:16.752650 5316 leader_election.cc:288] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [CANDIDATE]: Term 1 election: Requested vote from peers d4346c5843c646e4bbf1b621a5d0c875 (127.29.159.66:43979), cf85e20a1ec8466ca9bcd09b7df90878 (127.29.159.65:45531)
I20260927 18:47:16.752986 5021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa069db4d4ca43ef9ae927bac4dc36a3" candidate_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4346c5843c646e4bbf1b621a5d0c875"
I20260927 18:47:16.753015 4885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa069db4d4ca43ef9ae927bac4dc36a3" candidate_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf85e20a1ec8466ca9bcd09b7df90878"
I20260927 18:47:16.753115 5021 raft_consensus.cc:3037] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:16.753115 4885 raft_consensus.cc:3037] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:16.754292 4885 raft_consensus.cc:2454] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fae20bb731f4ce2bea52eaa0418c7d1 in term 1.
I20260927 18:47:16.754302 5021 raft_consensus.cc:2454] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fae20bb731f4ce2bea52eaa0418c7d1 in term 1.
I20260927 18:47:16.754558 5093 leader_election.cc:302] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fae20bb731f4ce2bea52eaa0418c7d1, d4346c5843c646e4bbf1b621a5d0c875; no voters:
I20260927 18:47:16.754750 5316 raft_consensus.cc:2781] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:16.754968 5316 raft_consensus.cc:686] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 1 LEADER]: Becoming Leader. State: Replica: 7fae20bb731f4ce2bea52eaa0418c7d1, State: Running, Role: LEADER
I20260927 18:47:16.755450 5316 consensus_queue.cc:237] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } } peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } }
I20260927 18:47:16.755684 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.755443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.67:47595)
0927 18:47:16.755544 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.755614 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:47:16.758129 4567 catalog_manager.cc:5495] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fae20bb731f4ce2bea52eaa0418c7d1 (127.29.159.67). New cstate: current_term: 1 leader_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fae20bb731f4ce2bea52eaa0418c7d1" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 } health_report { overall_health: UNKNOWN } } }
I20260927 18:47:16.803856 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.800494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39672 (local address 127.29.159.67:35793)
0927 18:47:16.800614 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:16.800618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:16.800631 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:47:16.800813 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:16.800815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:16.800872 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:16.801018 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:47:16.801224 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.801715 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.802555 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:16.802769 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:16.802843 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:16.803066 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:47:16.803521 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:47:16.803604 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:16.803697 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 18:47:16.803751 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:47:16.806594 5021 raft_consensus.cc:1261] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875 [term 1 FOLLOWER]: Refusing update from remote peer 7fae20bb731f4ce2bea52eaa0418c7d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:16.806861 4885 raft_consensus.cc:1261] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878 [term 1 FOLLOWER]: Refusing update from remote peer 7fae20bb731f4ce2bea52eaa0418c7d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:16.807166 5316 consensus_queue.cc:1035] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4346c5843c646e4bbf1b621a5d0c875" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:16.807466 5319 consensus_queue.cc:1035] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf85e20a1ec8466ca9bcd09b7df90878" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:16.809201 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.808927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.65:41437)
0927 18:47:16.809023 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.809114 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:47:16.809477 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.809191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.66:48333)
0927 18:47:16.809340 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.809411 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 18:47:16.812805 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.812021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.67:45243)
0927 18:47:16.812672 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.812733 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582}
I20260927 18:47:16.814108 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.813735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.66:37001)
0927 18:47:16.813981 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.814039 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260927 18:47:16.814199 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.813953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.67:37847)
0927 18:47:16.814080 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.814127 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 18:47:16.815685 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.815316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:40221 (local address 127.29.159.65:49859)
0927 18:47:16.815563 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.815614 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:40221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260927 18:47:16.820230 4568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:16.820390 4568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:16.824396 4568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59f72f81f60a469c8b9fdae0433cc518: Sending DeleteTablet for 3 replicas of tablet aa069db4d4ca43ef9ae927bac4dc36a3
I20260927 18:47:16.825026 5002 tablet_service.cc:1514] Processing DeleteTablet for tablet aa069db4d4ca43ef9ae927bac4dc36a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:16 UTC) from {username='oryx', principal='oryx/127.29.159.125@KRBTEST.COM'} at 127.0.0.1:57786
I20260927 18:47:16.825148 5138 tablet_service.cc:1514] Processing DeleteTablet for tablet aa069db4d4ca43ef9ae927bac4dc36a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:16 UTC) from {username='oryx', principal='oryx/127.29.159.125@KRBTEST.COM'} at 127.0.0.1:39662
I20260927 18:47:16.825258 4865 tablet_service.cc:1514] Processing DeleteTablet for tablet aa069db4d4ca43ef9ae927bac4dc36a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:16 UTC) from {username='oryx', principal='oryx/127.29.159.125@KRBTEST.COM'} at 127.0.0.1:51216
I20260927 18:47:16.825492 5333 tablet_replica.cc:330] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1: stopping tablet replica
I20260927 18:47:16.825593 5334 tablet_replica.cc:330] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878: stopping tablet replica
I20260927 18:47:16.825685 5333 raft_consensus.cc:2229] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:47:16.825814 5334 raft_consensus.cc:2229] T aa069db4d4ca43ef9ae927bac4dc36a3 P cf85e20a1ec8466ca9bcd09b7df90878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:16.826018 5333 raft_consensus.cc:2258] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:16.826128 5332 tablet_replica.cc:330] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875: stopping tablet replica
I20260927 18:47:16.826292 5332 raft_consensus.cc:2229] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:16.826398 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 4712
I20260927 18:47:16.826481 5332 raft_consensus.cc:2258] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:16.827804 5333 ts_tablet_manager.cc:1905] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:16.827804 5332 ts_tablet_manager.cc:1905] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:16.830447 5332 ts_tablet_manager.cc:1918] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:16.830523 5332 log.cc:1196] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/aa069db4d4ca43ef9ae927bac4dc36a3
I20260927 18:47:16.830615 5333 ts_tablet_manager.cc:1918] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:16.830688 5333 log.cc:1196] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/aa069db4d4ca43ef9ae927bac4dc36a3
I20260927 18:47:16.830816 5332 ts_tablet_manager.cc:1939] T aa069db4d4ca43ef9ae927bac4dc36a3 P d4346c5843c646e4bbf1b621a5d0c875: Deleting consensus metadata
I20260927 18:47:16.830962 5333 ts_tablet_manager.cc:1939] T aa069db4d4ca43ef9ae927bac4dc36a3 P 7fae20bb731f4ce2bea52eaa0418c7d1: Deleting consensus metadata
I20260927 18:47:16.831609 4552 catalog_manager.cc:4841] TS 7fae20bb731f4ce2bea52eaa0418c7d1 (127.29.159.67:35793): tablet aa069db4d4ca43ef9ae927bac4dc36a3 (table test-table [id=a1e4a17ccf38441fa2a3ac7415e8820f]) successfully deleted
I20260927 18:47:16.831609 4554 catalog_manager.cc:4841] TS d4346c5843c646e4bbf1b621a5d0c875 (127.29.159.66:43979): tablet aa069db4d4ca43ef9ae927bac4dc36a3 (table test-table [id=a1e4a17ccf38441fa2a3ac7415e8820f]) successfully deleted
I20260927 18:47:16.838294 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 4939
I20260927 18:47:16.840638 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:16.840318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:45531 (local address 127.0.0.1:51220)
0927 18:47:16.840481 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:16.840543 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 18:47:16.840960 4552 catalog_manager.cc:4584] TS cf85e20a1ec8466ca9bcd09b7df90878 (127.29.159.65:45531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa069db4d4ca43ef9ae927bac4dc36a3: Network error: Client connection negotiation failed: client connection to 127.29.159.65:45531: connect: Connection refused (error 111)
I20260927 18:47:16.844133 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 5075
I20260927 18:47:16.850001 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 4432
I20260927 18:47:16.856158 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 4539
2026-09-27T18:47:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7266 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:47:16 dist-test-slave-8tg2 krb5kdc[5337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:47:16 dist-test-slave-8tg2 krb5kdc[5337](info): set up 2 sockets
Sep 27 18:47:16 dist-test-slave-8tg2 krb5kdc[5337](info): commencing operation
krb5kdc: starting...
W20260927 18:47:17.908104 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:47:17 dist-test-slave-8tg2 krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:47:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.126.keytab.
Entry for principal oryx/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.126.keytab.
I20260927 18:47:18.030522 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:38113
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.126.keytab
--principal=oryx/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:38113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:18.138971 5350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:18.139288 5350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:18.139366 5350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:18.143325 5350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:18.143422 5350 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:47:18.143476 5350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:18.143541 5350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:18.143599 5350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:18.143635 5350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:18.148891 5350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:38113
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.126.keytab
--principal=oryx/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:38113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:18.149971 5350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:18.150986 5350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:18.159246 5356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:18.159317 5355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:18.159196 5358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:18.159482 5350 server_base.cc:1034] running on GCE node
I20260927 18:47:18.160040 5350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:18.161154 5350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:18.162359 5350 hybrid_clock.cc:648] HybridClock initialized: now 1790534838162340 us; error 31 us; skew 500 ppm
Sep 27 18:47:18 dist-test-slave-8tg2 krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534838, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:18.171530 5350 init.cc:377] Logged in from keytab as oryx/127.29.159.126@KRBTEST.COM (short username oryx)
I20260927 18:47:18.172395 5350 webserver.cc:460] Webserver started at http://127.29.159.126:37853/ using document root <none> and password file <none>
I20260927 18:47:18.172915 5350 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:18.172974 5350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:18.173416 5350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:18.175561 5350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "58af39a3b8024e46a80ff3482c28f444"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "189f12c23c7e5ad636a8f744fb4ae2fe"
server_key_iv: "d9ce802f4096936eeb12141788559ba5"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.176169 5350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "58af39a3b8024e46a80ff3482c28f444"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "189f12c23c7e5ad636a8f744fb4ae2fe"
server_key_iv: "d9ce802f4096936eeb12141788559ba5"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.180155 5350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:47:18.182687 5365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:18.184032 5350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:18.184178 5350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "58af39a3b8024e46a80ff3482c28f444"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "189f12c23c7e5ad636a8f744fb4ae2fe"
server_key_iv: "d9ce802f4096936eeb12141788559ba5"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.184301 5350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:18.215939 5350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:18.219074 5350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:18.219291 5350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:18.227150 5350 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:38113
I20260927 18:47:18.227193 5427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:38113 every 8 connection(s)
I20260927 18:47:18.228129 5350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:18.231032 5428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:18.236896 5428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444: Bootstrap starting.
I20260927 18:47:18.237178 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 5350
I20260927 18:47:18.237308 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:18.237636 30333 external_mini_cluster.cc:1428] Setting key 32b538e8165470fc1c82dd6ed160c8d4
I20260927 18:47:18.239847 5428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:18.240638 5428 log.cc:824] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:18.242654 5428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444: No bootstrap required, opened a new log
Sep 27 18:47:18 dist-test-slave-8tg2 krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:18.247140 5428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58af39a3b8024e46a80ff3482c28f444" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38113 } }
I20260927 18:47:18.247485 5428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:18.247597 5428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58af39a3b8024e46a80ff3482c28f444, State: Initialized, Role: FOLLOWER
I20260927 18:47:18.248157 5428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58af39a3b8024e46a80ff3482c28f444" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38113 } }
I20260927 18:47:18.248340 5428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:18.248423 5428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:18.248528 5428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:18.249485 5428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58af39a3b8024e46a80ff3482c28f444" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38113 } }
I20260927 18:47:18.249902 5428 leader_election.cc:302] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58af39a3b8024e46a80ff3482c28f444; no voters:
I20260927 18:47:18.250257 5428 leader_election.cc:288] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:18.250360 5433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:18.250579 5433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [term 1 LEADER]: Becoming Leader. State: Replica: 58af39a3b8024e46a80ff3482c28f444, State: Running, Role: LEADER
I20260927 18:47:18.250943 5433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58af39a3b8024e46a80ff3482c28f444" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38113 } }
I20260927 18:47:18.251770 5428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:18.253393 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.239029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45386 (local address 127.29.159.126:38113)
0927 18:47:18.239654 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:18.239669 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:18.239704 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:47:18.240400 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:18.240419 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:18.240799 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:18.241176 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:18.242001 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.243010 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.244353 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.244880 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.245072 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.247643 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:18.247649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:18.247662 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:18.247688 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:18.249498 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.250052 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.250061 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.250068 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.250215 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.250578 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.250581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.250583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.251074 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:18.251271 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:18.251946 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 18:47:18.252279 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":146,"threads_started":1}
I20260927 18:47:18.254565 5434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58af39a3b8024e46a80ff3482c28f444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58af39a3b8024e46a80ff3482c28f444" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38113 } } }
I20260927 18:47:18.254577 5435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58af39a3b8024e46a80ff3482c28f444. Latest consensus state: current_term: 1 leader_uuid: "58af39a3b8024e46a80ff3482c28f444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58af39a3b8024e46a80ff3482c28f444" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38113 } } }
I20260927 18:47:18.254683 5434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:18.254704 5435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444 [sys.catalog]: This master's current role is: LEADER
W20260927 18:47:18.256980 5448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:47:18.257068 5448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:47:18.257141 5449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:18.257784 5449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:18.263612 5449 catalog_manager.cc:1347] Generated new cluster ID: ccbba1cfbfae48d3898cb4257c591fc7
I20260927 18:47:18.263692 5449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:18.284592 5449 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:18.285633 5449 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:18.300151 5449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444: Generated new TSK 0
I20260927 18:47:18.300875 5449 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab.
Entry for principal oryx/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab.
I20260927 18:47:18.334982 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:38113
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab
--principal=oryx/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:18.445843 5454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:18.446110 5454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:18.446164 5454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:18.450286 5454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:18.450349 5454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:18.450390 5454 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:47:18.450423 5454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:18.455873 5454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab
--principal=oryx/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.159.126:38113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:18.456867 5454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:18.457839 5454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:18.467372 5462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:18.467458 5454 server_base.cc:1034] running on GCE node
W20260927 18:47:18.467381 5459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:18.467383 5460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:18.468086 5454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:18.468765 5454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:18.469969 5454 hybrid_clock.cc:648] HybridClock initialized: now 1790534838469882 us; error 98 us; skew 500 ppm
Sep 27 18:47:18 dist-test-slave-8tg2 krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534838, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:18.472911 5454 init.cc:377] Logged in from keytab as oryx/127.29.159.65@KRBTEST.COM (short username oryx)
I20260927 18:47:18.473790 5454 webserver.cc:460] Webserver started at http://127.29.159.65:35085/ using document root <none> and password file <none>
I20260927 18:47:18.474352 5454 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:18.474447 5454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:18.474678 5454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:18.476771 5454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "aa67eb538a6f4b09b7a2c19c527780a4"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "ab32f40d52dbd391bbdbcf6f96e1fb05"
server_key_iv: "d3656df98f6a48ea4bc18bc44b89a0af"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.477361 5454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "aa67eb538a6f4b09b7a2c19c527780a4"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "ab32f40d52dbd391bbdbcf6f96e1fb05"
server_key_iv: "d3656df98f6a48ea4bc18bc44b89a0af"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.481225 5454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 18:47:18.483594 5469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:18.484830 5454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:47:18.484977 5454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "aa67eb538a6f4b09b7a2c19c527780a4"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "ab32f40d52dbd391bbdbcf6f96e1fb05"
server_key_iv: "d3656df98f6a48ea4bc18bc44b89a0af"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.485090 5454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:18.506832 5454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:18.509846 5454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:18.510042 5454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:18.511518 5454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:18.511584 5454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:18.511650 5454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:18.511731 5454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:47:18 dist-test-slave-8tg2 krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534838, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:18.522636 5454 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:36709
I20260927 18:47:18.522770 5584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:36709 every 8 connection(s)
I20260927 18:47:18.523736 5454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:18.525724 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.512259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:51861 (local address 127.29.159.126:38113)
0927 18:47:18.512391 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:18.512395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:18.513375 (+ 980us) server_negotiation.cc:407] Connection header received
0927 18:47:18.514356 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:18.514363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:18.514427 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:18.514527 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:18.515979 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.516566 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.518080 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.518319 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.518447 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.522093 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:18.522097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:18.522099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:18.522133 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:18.523717 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.524263 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.524266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.524268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.524353 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.524716 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.524719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.524720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.524900 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:18.524983 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:18.525485 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 18:47:18.525583 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:47:18.526502 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.512745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:38113 (local address 127.29.159.65:51861)
0927 18:47:18.513223 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:18.513266 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:18.514113 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:18.514851 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:18.514860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:18.515238 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:18.515814 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.515829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.516703 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.516707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.517943 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.517953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.518507 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.518512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.518713 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.519374 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:18.519400 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:18.521915 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:18.523838 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.523844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.523858 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.524110 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.524455 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.524458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.524461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.524609 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.525000 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:18.525005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:18.525285 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 18:47:18.525441 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 18:47:18.525672 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":133,"threads_started":1}
I20260927 18:47:18.527812 5585 heartbeater.cc:347] Connected to a master server at 127.29.159.126:38113
I20260927 18:47:18.527940 5585 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:18.528434 5585 heartbeater.cc:511] Master 127.29.159.126:38113 requested a full tablet report, sending...
I20260927 18:47:18.530118 5381 ts_manager.cc:194] Registered new tserver with Master: aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709)
I20260927 18:47:18.531327 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 5454
I20260927 18:47:18.531487 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:47:18.531621 5381 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:51861
I20260927 18:47:18.531806 30333 external_mini_cluster.cc:1428] Setting key 8118de2778f1f9bb91f1e545bccbd12f
I20260927 18:47:18.538236 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.529915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:38113 (local address 127.29.159.65:54593)
0927 18:47:18.530318 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:18.530333 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:18.530432 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:18.530657 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:18.530660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:18.530859 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:18.530974 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.530981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.532121 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.532124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.533028 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.533037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.533660 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.533666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.533792 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.534291 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:18.534308 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:18.534784 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:18.536686 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.536689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.536690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.536878 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.537211 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.537214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.537215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.537301 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.537752 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:18.537754 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:18.537824 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 18:47:18.537904 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 18:47:18.538007 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":97,"threads_started":1}
I20260927 18:47:18.538372 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.530025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:54593 (local address 127.29.159.126:38113)
0927 18:47:18.530166 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:18.530170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:18.530410 (+ 240us) server_negotiation.cc:407] Connection header received
0927 18:47:18.530512 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:18.530514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:18.530552 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:18.530614 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:18.531197 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.531973 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.533225 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.533521 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.533639 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.534947 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:18.534952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:18.534957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:18.534988 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:18.536557 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.536981 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.536988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.536992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.537083 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.537406 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.537412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.537417 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.537650 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:18.537755 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:18.537997 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 18:47:18.538149 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for oryx/127.29.159.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab.
Entry for principal oryx/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab.
I20260927 18:47:18.562058 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:38113
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab
--principal=oryx/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:18.671083 5594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:18.671329 5594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:18.671373 5594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:18.675277 5594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:18.675333 5594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:18.675371 5594 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:47:18.675403 5594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:18.680951 5594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab
--principal=oryx/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.159.126:38113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:18.681903 5594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:18.682960 5594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:18.692417 5599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:18.692428 5602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:18.692418 5600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:18.692889 5594 server_base.cc:1034] running on GCE node
I20260927 18:47:18.693290 5594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:18.693921 5594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:18.695168 5594 hybrid_clock.cc:648] HybridClock initialized: now 1790534838695156 us; error 30 us; skew 500 ppm
Sep 27 18:47:18 dist-test-slave-8tg2 krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534838, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:18.698050 5594 init.cc:377] Logged in from keytab as oryx/127.29.159.66@KRBTEST.COM (short username oryx)
I20260927 18:47:18.698956 5594 webserver.cc:460] Webserver started at http://127.29.159.66:36249/ using document root <none> and password file <none>
I20260927 18:47:18.699600 5594 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:18.699697 5594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:18.699929 5594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:18.701959 5594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "65aab0f618194c6e8d4cb4a89a165b12"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "f8433db863f5160061e2c55f2fcafeb6"
server_key_iv: "23af9e6f7cf633a8646c71c5c4ef5d75"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.702530 5594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "65aab0f618194c6e8d4cb4a89a165b12"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "f8433db863f5160061e2c55f2fcafeb6"
server_key_iv: "23af9e6f7cf633a8646c71c5c4ef5d75"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.706174 5594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 18:47:18.708670 5609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:18.709820 5594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:47:18.709954 5594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "65aab0f618194c6e8d4cb4a89a165b12"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "f8433db863f5160061e2c55f2fcafeb6"
server_key_iv: "23af9e6f7cf633a8646c71c5c4ef5d75"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.710083 5594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:18.732740 5594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:18.736236 5594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:18.736443 5594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:18.737865 5594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:18.737941 5594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:18.738011 5594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:18.738065 5594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:47:18 dist-test-slave-8tg2 krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534838, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:18.748883 5594 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:33253
I20260927 18:47:18.749007 5724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:33253 every 8 connection(s)
I20260927 18:47:18.750064 5594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:18.752354 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.738782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:47283 (local address 127.29.159.126:38113)
0927 18:47:18.738949 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:18.738955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:18.739767 (+ 812us) server_negotiation.cc:407] Connection header received
0927 18:47:18.740836 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:18.740840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:18.740903 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:18.740965 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:18.742829 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.743761 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.745153 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.745522 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.745636 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.748468 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:18.748473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:18.748477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:18.748511 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:18.750167 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.750763 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.750766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.750768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.750856 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.751230 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.751233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.751234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.751415 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:18.751552 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:18.752123 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 18:47:18.752229 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:47:18.753424 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.739145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:38113 (local address 127.29.159.66:47283)
0927 18:47:18.739611 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:18.739660 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:18.740548 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:18.741454 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:18.741465 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:18.741963 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:18.742632 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.742651 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.743923 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.743927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.745013 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.745023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.745689 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.745693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.745893 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.746495 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:18.746511 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:18.748282 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:18.750278 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.750283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.750294 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.750629 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.750965 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.750968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.750970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.751135 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.751562 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:18.751567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:18.751844 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 18:47:18.752088 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 18:47:18.752402 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":141,"threads_started":1}
I20260927 18:47:18.754741 5725 heartbeater.cc:347] Connected to a master server at 127.29.159.126:38113
I20260927 18:47:18.754933 5725 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:18.755507 5725 heartbeater.cc:511] Master 127.29.159.126:38113 requested a full tablet report, sending...
I20260927 18:47:18.756641 5381 ts_manager.cc:194] Registered new tserver with Master: 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253)
I20260927 18:47:18.757432 5381 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:47283
I20260927 18:47:18.758528 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 5594
I20260927 18:47:18.758667 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:18.759018 30333 external_mini_cluster.cc:1428] Setting key d269179249df3c2a4bc8ef7505e0d49c
I20260927 18:47:18.764799 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.756998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:41657 (local address 127.29.159.126:38113)
0927 18:47:18.757125 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:18.757129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:18.757280 (+ 151us) server_negotiation.cc:407] Connection header received
0927 18:47:18.757398 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:18.757400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:18.757435 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:18.757496 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:18.758030 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.758779 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.759850 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.760052 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.760169 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.761466 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:18.761469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:18.761471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:18.761493 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:18.763053 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.763521 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.763525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.763526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.763616 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.763909 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.763912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.763914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.764137 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:18.764265 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:18.764523 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 18:47:18.764634 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:47:18.764827 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.756938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:38113 (local address 127.29.159.66:41657)
0927 18:47:18.757212 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:18.757226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:18.757320 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:18.757541 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:18.757544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:18.757770 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:18.757893 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.757901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.758932 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.758935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.759747 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.759758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.760172 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.760175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.760279 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.760813 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:18.760828 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:18.761322 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:18.763174 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.763179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.763181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.763383 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.763723 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.763726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.763728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.763810 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.764256 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:18.764259 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:18.764359 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 18:47:18.764469 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 18:47:18.764625 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for oryx/127.29.159.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab.
Entry for principal oryx/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab.
I20260927 18:47:18.790261 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:38113
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab
--principal=oryx/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:18.904143 5734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:18.904443 5734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:18.904488 5734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:18.908620 5734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:18.908687 5734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:18.908726 5734 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:47:18.908802 5734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:18.914269 5734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab
--principal=oryx/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.159.126:38113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:18.915373 5734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:18.916539 5734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:18.926157 5740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:18.926187 5742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:18.926191 5739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:18.926448 5734 server_base.cc:1034] running on GCE node
I20260927 18:47:18.926942 5734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:18.927673 5734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:18.928817 5734 hybrid_clock.cc:648] HybridClock initialized: now 1790534838928786 us; error 60 us; skew 500 ppm
Sep 27 18:47:18 dist-test-slave-8tg2 krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534838, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:18.931774 5734 init.cc:377] Logged in from keytab as oryx/127.29.159.67@KRBTEST.COM (short username oryx)
I20260927 18:47:18.932718 5734 webserver.cc:460] Webserver started at http://127.29.159.67:32855/ using document root <none> and password file <none>
I20260927 18:47:18.933315 5734 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:18.933411 5734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:18.933626 5734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:18.935788 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "6c1978810c554fc09c4f9682638a0263"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "1892cb752f2a11a7f6a93ed2ca4421f1"
server_key_iv: "f1aeaa044a5ce76efa1aec49226a1da7"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.936375 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "6c1978810c554fc09c4f9682638a0263"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "1892cb752f2a11a7f6a93ed2ca4421f1"
server_key_iv: "f1aeaa044a5ce76efa1aec49226a1da7"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.940320 5734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:47:18.942880 5749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:18.944170 5734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:18.944310 5734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "6c1978810c554fc09c4f9682638a0263"
format_stamp: "Formatted at 2026-09-27 18:47:18 on dist-test-slave-8tg2"
server_key: "1892cb752f2a11a7f6a93ed2ca4421f1"
server_key_iv: "f1aeaa044a5ce76efa1aec49226a1da7"
server_key_version: "encryptionkey@0"
I20260927 18:47:18.944430 5734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:18.963610 5734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:18.967046 5734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:18.967249 5734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:18.968757 5734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:18.968832 5734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:18.968917 5734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:18.968967 5734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:47:18 dist-test-slave-8tg2 krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534838, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:18.980288 5734 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:37917
I20260927 18:47:18.980470 5864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:37917 every 8 connection(s)
I20260927 18:47:18.981395 5734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:18.983796 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.969826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:58505 (local address 127.29.159.126:38113)
0927 18:47:18.969953 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:18.969957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:18.971014 (+ 1057us) server_negotiation.cc:407] Connection header received
0927 18:47:18.972189 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:18.972193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:18.972250 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:18.972336 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:18.973769 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.974437 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.975808 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.976119 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.976240 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.979650 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:18.979654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:18.979657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:18.979692 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:18.981458 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.982073 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.982077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.982079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.982260 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.982638 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.982641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.982643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.982845 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:18.982930 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:18.983553 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 18:47:18.983666 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:47:18.984525 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.970363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:38113 (local address 127.29.159.67:58505)
0927 18:47:18.970859 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:18.970901 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:18.971933 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:18.972776 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:18.972783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:18.973101 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:18.973631 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.973644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.974555 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.974558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.975689 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.975698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.976293 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.976296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.976470 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.977082 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:18.977106 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:18.979415 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:18.981593 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.981600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.981615 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.981940 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.982392 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.982394 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.982396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.982524 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.982937 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:18.982942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:18.983260 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 18:47:18.983410 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 18:47:18.983723 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":139,"threads_started":1}
I20260927 18:47:18.985661 5865 heartbeater.cc:347] Connected to a master server at 127.29.159.126:38113
I20260927 18:47:18.985814 5865 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:18.986286 5865 heartbeater.cc:511] Master 127.29.159.126:38113 requested a full tablet report, sending...
I20260927 18:47:18.987103 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 5734
I20260927 18:47:18.987274 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:18.987391 5381 ts_manager.cc:194] Registered new tserver with Master: 6c1978810c554fc09c4f9682638a0263 (127.29.159.67:37917)
I20260927 18:47:18.987665 30333 external_mini_cluster.cc:1428] Setting key 32b8e15f05003b8ddc8314f8e06e0bdb
I20260927 18:47:18.988267 5381 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:58505
I20260927 18:47:18.990221 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:47:18.997236 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.988290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:43641 (local address 127.29.159.126:38113)
0927 18:47:18.988565 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:18.988570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:18.988607 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:47:18.988800 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:18.988804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:18.988848 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:18.988901 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:18.989507 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.991611 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.992412 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.992640 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.992757 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.993813 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:18.993817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:18.993819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:18.993848 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:18.995497 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.995963 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.995970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.995975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.996070 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:18.996350 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:18.996356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:18.996361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:18.996605 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:18.996709 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:18.996965 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 18:47:18.997086 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260927 18:47:18.997252 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:18.988239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:38113 (local address 127.29.159.67:43641)
0927 18:47:18.988538 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:18.988555 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:18.988649 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:18.989065 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:18.989068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:18.989279 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:18.989389 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.989395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.991730 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.991733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.992292 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:18.992300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:18.992751 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:18.992754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:18.992857 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:18.993359 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:18.993370 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:18.993679 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:18.995652 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.995655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.995657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.995862 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.996178 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:18.996180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:18.996182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:18.996259 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:18.996700 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:18.996704 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:18.996800 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 18:47:18.996911 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 18:47:18.997050 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":115,"threads_started":1}
Sep 27 18:47:18 dist-test-slave-8tg2 krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:47:19 dist-test-slave-8tg2 krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:19 dist-test-slave-8tg2 krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:47:19.016155 5431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:19.009756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.29.159.126:38113)
0927 18:47:19.009882 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.009885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.009898 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:47:19.009939 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.009942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.009981 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.010055 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:19.010834 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.011346 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.012342 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.012549 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.012629 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.015997 (+ 3368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45394: BlockingRecv error: recv got EOF from 127.0.0.1:45394 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 18:47:19 dist-test-slave-8tg2 krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:19.026151 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.017775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.29.159.126:38113)
0927 18:47:19.017882 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.017885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.018009 (+ 124us) server_negotiation.cc:407] Connection header received
0927 18:47:19.018181 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.018185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.018229 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.018316 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:19.018904 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.019602 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.020398 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.020621 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.020706 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.022890 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:19.022893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:19.022896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:19.022917 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:19.024455 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:19.024852 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:19.024855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:19.024856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:19.024925 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:19.025156 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:19.025158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:19.025159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:19.025417 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:19.025569 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.025830 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.025964 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:47:19.029081 5382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:47:19.032171 5382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:47:19.046697 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.040419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:37917 (local address 127.0.0.1:58582)
0927 18:47:19.040771 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:19.040827 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:19.040949 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:19.041889 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:19.041894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:19.041916 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:19.042063 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.042071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.043567 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.043570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.045008 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.045016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.046137 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.046141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.046294 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.046336 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 18:47:19.046444 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 18:47:19.046571 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":176,"thread_start_us":108,"threads_started":1}
I20260927 18:47:19.046892 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.041137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:36709 (local address 127.0.0.1:48968)
0927 18:47:19.041653 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:19.041667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:19.041759 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:19.042609 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:19.042612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:19.042626 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:19.042739 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.042745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.043960 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.043963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.045318 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.045329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.046369 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.046372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.046518 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.046536 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:47:19.046601 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:47:19.046669 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":168,"thread_start_us":100,"threads_started":1}
I20260927 18:47:19.046973 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.041297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.29.159.65:36709)
0927 18:47:19.042185 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.042193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.042217 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:47:19.042287 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.042292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.042470 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.042856 (+ 386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:19.042954 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.043826 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.045452 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.046227 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.046535 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.046682 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.046734 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.046840 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":70,"threads_started":1}
I20260927 18:47:19.047312 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.040713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:33253 (local address 127.0.0.1:40298)
0927 18:47:19.041557 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:19.041570 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:19.041666 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:19.042484 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:19.042487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:19.042503 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:19.042611 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.042617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.043915 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.043921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.045318 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.045329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.046331 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.046338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.046444 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.046735 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 18:47:19.047155 (+ 420us) client_negotiation.cc:240] Negotiation successful
0927 18:47:19.047225 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":98,"thread_start_us":86,"threads_started":1}
I20260927 18:47:19.047364 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.040603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.29.159.67:37917)
0927 18:47:19.041333 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.041348 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.041373 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:47:19.041451 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.041458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.041724 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.041903 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:19.042255 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.043470 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.045131 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.045872 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.045982 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.046142 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.047133 (+ 991us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.047231 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":116,"threads_started":1}
I20260927 18:47:19.047861 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.041231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.29.159.66:33253)
0927 18:47:19.042014 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.042026 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.042048 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:47:19.042123 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.042127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.042345 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.042492 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:19.042728 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.043807 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.045463 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.046202 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.046306 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.046460 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.047603 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.047717 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":97,"threads_started":1}
I20260927 18:47:19.049949 5519 tablet_service.cc:1467] Processing CreateTablet for tablet 5625a73c9b8a46b2915f6aec12eda083 (DEFAULT_TABLE table=test-table [id=027c44719121460b9a29434bf8b033ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.050040 5799 tablet_service.cc:1467] Processing CreateTablet for tablet 5625a73c9b8a46b2915f6aec12eda083 (DEFAULT_TABLE table=test-table [id=027c44719121460b9a29434bf8b033ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.050328 5659 tablet_service.cc:1467] Processing CreateTablet for tablet 5625a73c9b8a46b2915f6aec12eda083 (DEFAULT_TABLE table=test-table [id=027c44719121460b9a29434bf8b033ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.050858 5519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5625a73c9b8a46b2915f6aec12eda083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.051030 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5625a73c9b8a46b2915f6aec12eda083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.051108 5659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5625a73c9b8a46b2915f6aec12eda083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.057518 5893 tablet_bootstrap.cc:492] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4: Bootstrap starting.
I20260927 18:47:19.059500 5894 tablet_bootstrap.cc:492] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12: Bootstrap starting.
I20260927 18:47:19.059739 5895 tablet_bootstrap.cc:492] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263: Bootstrap starting.
I20260927 18:47:19.059831 5893 tablet_bootstrap.cc:654] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.060704 5893 log.cc:824] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:19.061523 5894 tablet_bootstrap.cc:654] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.061523 5895 tablet_bootstrap.cc:654] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.062368 5894 log.cc:824] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:19.062367 5895 log.cc:824] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:19.062489 5893 tablet_bootstrap.cc:492] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4: No bootstrap required, opened a new log
I20260927 18:47:19.062666 5893 ts_tablet_manager.cc:1397] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:47:19.064337 5895 tablet_bootstrap.cc:492] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263: No bootstrap required, opened a new log
I20260927 18:47:19.064513 5895 ts_tablet_manager.cc:1397] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:47:19.064888 5894 tablet_bootstrap.cc:492] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12: No bootstrap required, opened a new log
I20260927 18:47:19.065059 5894 ts_tablet_manager.cc:1397] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:47:19.066864 5893 raft_consensus.cc:348] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } }
I20260927 18:47:19.067112 5893 raft_consensus.cc:374] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.067165 5893 raft_consensus.cc:729] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa67eb538a6f4b09b7a2c19c527780a4, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.067615 5893 consensus_queue.cc:260] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } }
I20260927 18:47:19.068326 5585 heartbeater.cc:503] Master 127.29.159.126:38113 was elected leader, sending a full tablet report...
I20260927 18:47:19.068411 5895 raft_consensus.cc:348] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } }
I20260927 18:47:19.068632 5893 ts_tablet_manager.cc:1428] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 18:47:19.068652 5895 raft_consensus.cc:374] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.068711 5895 raft_consensus.cc:729] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1978810c554fc09c4f9682638a0263, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.068748 5894 raft_consensus.cc:348] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } }
I20260927 18:47:19.068969 5894 raft_consensus.cc:374] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.069015 5894 raft_consensus.cc:729] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65aab0f618194c6e8d4cb4a89a165b12, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.069176 5895 consensus_queue.cc:260] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } }
I20260927 18:47:19.069406 5894 consensus_queue.cc:260] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } }
I20260927 18:47:19.070159 5895 ts_tablet_manager.cc:1428] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:47:19.070246 5894 ts_tablet_manager.cc:1428] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:47:19.070541 5865 heartbeater.cc:503] Master 127.29.159.126:38113 was elected leader, sending a full tablet report...
I20260927 18:47:19.071395 5725 heartbeater.cc:503] Master 127.29.159.126:38113 was elected leader, sending a full tablet report...
I20260927 18:47:19.169646 5900 raft_consensus.cc:482] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:19.169891 5900 raft_consensus.cc:504] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } }
I20260927 18:47:19.171099 5900 leader_election.cc:288] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709), 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253)
I20260927 18:47:19.175051 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.171413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:39265 (local address 127.29.159.65:36709)
0927 18:47:19.171593 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.171597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.171615 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:47:19.171674 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.171678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.171730 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.171834 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:19.172187 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.172784 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.173909 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.174511 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.174628 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.174689 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.174899 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.174956 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 18:47:19.175074 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.171313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:36709 (local address 127.29.159.67:39265)
0927 18:47:19.171450 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:19.171469 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:19.171572 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:19.171870 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:19.171875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:19.171898 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:19.172047 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.172054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.172898 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.172902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.173785 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.173794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.174631 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.174634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.174762 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.174788 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:47:19.174876 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:47:19.174945 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 18:47:19.175760 5539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625a73c9b8a46b2915f6aec12eda083" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" is_pre_election: true
I20260927 18:47:19.175874 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.171409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:33253 (local address 127.29.159.67:60941)
0927 18:47:19.171811 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:19.171826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:19.171919 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:19.172183 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:19.172186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:19.172202 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:19.172331 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.172337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.173122 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.173125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.174632 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.174642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.175564 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.175567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.175637 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.175653 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 18:47:19.175716 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 18:47:19.175772 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"spinlock_wait_cycles":1408,"thread_start_us":113,"threads_started":1}
I20260927 18:47:19.175940 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.171675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:60941 (local address 127.29.159.66:33253)
0927 18:47:19.171841 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.171846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.171882 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:47:19.172035 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.172038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.172085 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.172162 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:19.172439 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.173012 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.174754 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.175357 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.175524 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.175569 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.175757 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.175826 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:47:19.176189 5539 raft_consensus.cc:2454] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 0.
I20260927 18:47:19.176589 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625a73c9b8a46b2915f6aec12eda083" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65aab0f618194c6e8d4cb4a89a165b12" is_pre_election: true
I20260927 18:47:19.176748 5750 leader_election.cc:302] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1978810c554fc09c4f9682638a0263, aa67eb538a6f4b09b7a2c19c527780a4; no voters:
I20260927 18:47:19.176904 5679 raft_consensus.cc:2454] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 0.
I20260927 18:47:19.177080 5900 raft_consensus.cc:2781] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:19.177176 5900 raft_consensus.cc:482] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:19.177239 5900 raft_consensus.cc:3037] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.178305 5900 raft_consensus.cc:504] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } }
I20260927 18:47:19.178714 5900 leader_election.cc:288] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 election: Requested vote from peers aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709), 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253)
I20260927 18:47:19.179013 5539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625a73c9b8a46b2915f6aec12eda083" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67eb538a6f4b09b7a2c19c527780a4"
I20260927 18:47:19.179123 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5625a73c9b8a46b2915f6aec12eda083" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65aab0f618194c6e8d4cb4a89a165b12"
I20260927 18:47:19.179152 5539 raft_consensus.cc:3037] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.179243 5679 raft_consensus.cc:3037] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.180209 5539 raft_consensus.cc:2454] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 1.
I20260927 18:47:19.180305 5679 raft_consensus.cc:2454] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 1.
I20260927 18:47:19.180491 5750 leader_election.cc:302] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1978810c554fc09c4f9682638a0263, aa67eb538a6f4b09b7a2c19c527780a4; no voters:
I20260927 18:47:19.180647 5900 raft_consensus.cc:2781] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:19.180897 5900 raft_consensus.cc:686] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 1 LEADER]: Becoming Leader. State: Replica: 6c1978810c554fc09c4f9682638a0263, State: Running, Role: LEADER
I20260927 18:47:19.181214 5900 consensus_queue.cc:237] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } }
I20260927 18:47:19.184350 5381 catalog_manager.cc:5495] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c1978810c554fc09c4f9682638a0263 (127.29.159.67). New cstate: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } health_report { overall_health: UNKNOWN } } }
I20260927 18:47:19.201587 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.198347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.29.159.67:37917)
0927 18:47:19.198473 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.198476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.198490 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:47:19.198640 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.198642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.198692 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.198781 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:47:19.198983 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.199544 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.200354 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.200565 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.200640 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.200878 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:47:19.201283 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:47:19.201376 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.201431 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.201478 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:47:19.206974 5679 raft_consensus.cc:1261] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Refusing update from remote peer 6c1978810c554fc09c4f9682638a0263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:19.206974 5539 raft_consensus.cc:1261] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c1978810c554fc09c4f9682638a0263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:19.207655 5903 consensus_queue.cc:1035] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:19.207837 5900 consensus_queue.cc:1035] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:19.218801 5908 mvcc.cc:205] Tried to move back new op lower bound from 7334030701385240576 to 7334030701288665088. Current Snapshot: MvccSnapshot[applied={T|T < 7334030701385240576}]
I20260927 18:47:19.223171 5380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:19.223342 5380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:19.225756 5380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444: Sending DeleteTablet for 3 replicas of tablet 5625a73c9b8a46b2915f6aec12eda083
I20260927 18:47:19.226677 5659 tablet_service.cc:1514] Processing DeleteTablet for tablet 5625a73c9b8a46b2915f6aec12eda083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:40298
I20260927 18:47:19.226670 5799 tablet_service.cc:1514] Processing DeleteTablet for tablet 5625a73c9b8a46b2915f6aec12eda083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:58582
I20260927 18:47:19.227156 5519 tablet_service.cc:1514] Processing DeleteTablet for tablet 5625a73c9b8a46b2915f6aec12eda083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:48968
I20260927 18:47:19.227138 5380 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:47:19.227617 5380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:47:19.227788 5915 tablet_replica.cc:330] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263: stopping tablet replica
I20260927 18:47:19.228071 5915 raft_consensus.cc:2229] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:47:19.228474 5915 raft_consensus.cc:2258] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.229952 5915 ts_tablet_manager.cc:1905] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.232191 5519 tablet_service.cc:1467] Processing CreateTablet for tablet a7f9268194fd4afbb1dba818be366900 (DEFAULT_TABLE table=test-table [id=5bdf2b6ea56943e9ad4878a292b183ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.232451 5519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f9268194fd4afbb1dba818be366900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.232515 5917 tablet_replica.cc:330] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4: stopping tablet replica
I20260927 18:47:19.232650 5915 ts_tablet_manager.cc:1918] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:19.232726 5915 log.cc:1196] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/5625a73c9b8a46b2915f6aec12eda083
I20260927 18:47:19.232832 5917 raft_consensus.cc:2229] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:19.233036 5915 ts_tablet_manager.cc:1939] T 5625a73c9b8a46b2915f6aec12eda083 P 6c1978810c554fc09c4f9682638a0263: Deleting consensus metadata
I20260927 18:47:19.233135 5917 raft_consensus.cc:2258] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.234210 5366 catalog_manager.cc:4841] TS 6c1978810c554fc09c4f9682638a0263 (127.29.159.67:37917): tablet 5625a73c9b8a46b2915f6aec12eda083 (table test-table [id=027c44719121460b9a29434bf8b033ac]) successfully deleted
I20260927 18:47:19.234527 5916 tablet_replica.cc:330] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12: stopping tablet replica
I20260927 18:47:19.234579 5917 ts_tablet_manager.cc:1905] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.234759 5893 tablet_bootstrap.cc:492] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4: Bootstrap starting.
I20260927 18:47:19.234853 5916 raft_consensus.cc:2229] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:19.234954 5799 tablet_service.cc:1467] Processing CreateTablet for tablet a7f9268194fd4afbb1dba818be366900 (DEFAULT_TABLE table=test-table [id=5bdf2b6ea56943e9ad4878a292b183ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.235139 5916 raft_consensus.cc:2258] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.235164 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f9268194fd4afbb1dba818be366900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.237242 5895 tablet_bootstrap.cc:492] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263: Bootstrap starting.
I20260927 18:47:19.238184 5895 tablet_bootstrap.cc:654] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.238291 5659 tablet_service.cc:1467] Processing CreateTablet for tablet a7f9268194fd4afbb1dba818be366900 (DEFAULT_TABLE table=test-table [id=5bdf2b6ea56943e9ad4878a292b183ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.238689 5659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f9268194fd4afbb1dba818be366900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.240379 5916 ts_tablet_manager.cc:1905] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.241313 5894 tablet_bootstrap.cc:492] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12: Bootstrap starting.
I20260927 18:47:19.242203 5894 tablet_bootstrap.cc:654] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.243726 5894 tablet_bootstrap.cc:492] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12: No bootstrap required, opened a new log
I20260927 18:47:19.243810 5894 ts_tablet_manager.cc:1397] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:19.244038 5916 ts_tablet_manager.cc:1918] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:19.244186 5916 log.cc:1196] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/5625a73c9b8a46b2915f6aec12eda083
I20260927 18:47:19.244259 5894 raft_consensus.cc:348] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.244387 5894 raft_consensus.cc:374] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.244427 5894 raft_consensus.cc:729] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65aab0f618194c6e8d4cb4a89a165b12, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.244545 5894 consensus_queue.cc:260] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.244657 5916 ts_tablet_manager.cc:1939] T 5625a73c9b8a46b2915f6aec12eda083 P 65aab0f618194c6e8d4cb4a89a165b12: Deleting consensus metadata
I20260927 18:47:19.244773 5894 ts_tablet_manager.cc:1428] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:19.246474 5369 catalog_manager.cc:4841] TS 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253): tablet 5625a73c9b8a46b2915f6aec12eda083 (table test-table [id=027c44719121460b9a29434bf8b033ac]) successfully deleted
I20260927 18:47:19.246950 5895 tablet_bootstrap.cc:492] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263: No bootstrap required, opened a new log
I20260927 18:47:19.247040 5895 ts_tablet_manager.cc:1397] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260927 18:47:19.247525 5893 tablet_bootstrap.cc:654] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.247623 5895 raft_consensus.cc:348] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.247759 5895 raft_consensus.cc:374] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.247800 5895 raft_consensus.cc:729] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1978810c554fc09c4f9682638a0263, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.247916 5895 consensus_queue.cc:260] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.248186 5895 ts_tablet_manager.cc:1428] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:19.249249 5917 ts_tablet_manager.cc:1918] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:19.249349 5917 log.cc:1196] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/wal/wals/5625a73c9b8a46b2915f6aec12eda083
I20260927 18:47:19.249667 5917 ts_tablet_manager.cc:1939] T 5625a73c9b8a46b2915f6aec12eda083 P aa67eb538a6f4b09b7a2c19c527780a4: Deleting consensus metadata
I20260927 18:47:19.250845 5366 catalog_manager.cc:4841] TS aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709): tablet 5625a73c9b8a46b2915f6aec12eda083 (table test-table [id=027c44719121460b9a29434bf8b033ac]) successfully deleted
W20260927 18:47:19.251276 5726 tablet.cc:2369] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:19.251519 5893 tablet_bootstrap.cc:492] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4: No bootstrap required, opened a new log
I20260927 18:47:19.251624 5893 ts_tablet_manager.cc:1397] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20260927 18:47:19.252354 5893 raft_consensus.cc:348] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.252473 5893 raft_consensus.cc:374] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.252518 5893 raft_consensus.cc:729] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa67eb538a6f4b09b7a2c19c527780a4, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.252662 5893 consensus_queue.cc:260] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.252921 5893 ts_tablet_manager.cc:1428] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260927 18:47:19.275177 5586 tablet.cc:2369] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:19.284623 5903 raft_consensus.cc:482] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:19.284801 5903 raft_consensus.cc:504] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.285310 5903 leader_election.cc:288] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709), 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253)
I20260927 18:47:19.285749 5539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f9268194fd4afbb1dba818be366900" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" is_pre_election: true
I20260927 18:47:19.285804 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f9268194fd4afbb1dba818be366900" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65aab0f618194c6e8d4cb4a89a165b12" is_pre_election: true
I20260927 18:47:19.285924 5539 raft_consensus.cc:2454] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 0.
I20260927 18:47:19.285969 5679 raft_consensus.cc:2454] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 0.
I20260927 18:47:19.286248 5750 leader_election.cc:302] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1978810c554fc09c4f9682638a0263, aa67eb538a6f4b09b7a2c19c527780a4; no voters:
I20260927 18:47:19.286433 5903 raft_consensus.cc:2781] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:19.286542 5903 raft_consensus.cc:482] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:19.286613 5903 raft_consensus.cc:3037] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.287859 5903 raft_consensus.cc:504] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.288251 5903 leader_election.cc:288] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 election: Requested vote from peers aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709), 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253)
I20260927 18:47:19.288592 5539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f9268194fd4afbb1dba818be366900" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67eb538a6f4b09b7a2c19c527780a4"
I20260927 18:47:19.288631 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f9268194fd4afbb1dba818be366900" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65aab0f618194c6e8d4cb4a89a165b12"
I20260927 18:47:19.288717 5539 raft_consensus.cc:3037] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.288777 5679 raft_consensus.cc:3037] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.289861 5539 raft_consensus.cc:2454] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 1.
I20260927 18:47:19.290004 5679 raft_consensus.cc:2454] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 1.
I20260927 18:47:19.290169 5750 leader_election.cc:302] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1978810c554fc09c4f9682638a0263, aa67eb538a6f4b09b7a2c19c527780a4; no voters:
I20260927 18:47:19.290319 5903 raft_consensus.cc:2781] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:19.290386 5903 raft_consensus.cc:686] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 1 LEADER]: Becoming Leader. State: Replica: 6c1978810c554fc09c4f9682638a0263, State: Running, Role: LEADER
I20260927 18:47:19.290508 5903 consensus_queue.cc:237] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.292121 5380 catalog_manager.cc:5495] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c1978810c554fc09c4f9682638a0263 (127.29.159.67). New cstate: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } health_report { overall_health: HEALTHY } } }
Sep 27 18:47:19 dist-test-slave-8tg2 krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534838, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.126@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:19.342432 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.333097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:38113 (local address 127.0.0.1:45416)
0927 18:47:19.333430 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:19.333445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:19.333555 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:19.333833 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:19.333836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:19.334209 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:19.334323 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.334329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.335403 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.335405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.336038 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.336047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.336626 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.336629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.336692 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.337326 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:19.337338 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:19.338802 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:19.340803 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:19.340812 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:19.340817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:19.341076 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:19.341399 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:19.341402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:19.341404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:19.341497 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:19.341961 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:19.341966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:19.342069 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 18:47:19.342159 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 18:47:19.342293 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":120,"threads_started":1}
I20260927 18:47:19.342573 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.333215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.29.159.126:38113)
0927 18:47:19.333379 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.333384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.333527 (+ 143us) server_negotiation.cc:407] Connection header received
0927 18:47:19.333657 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.333660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.333712 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.333798 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:19.334448 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.335284 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.336159 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.336507 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.336624 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.338960 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:19.338964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:19.338967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:19.338992 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:19.340689 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:19.341184 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:19.341188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:19.341190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:19.341282 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:19.341616 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:19.341619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:19.341620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:19.341850 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:19.341953 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.342252 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.342374 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:47:19.346489 5380 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:45416:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:47:19.347239 5380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:47:19.352367 5659 tablet_service.cc:1467] Processing CreateTablet for tablet 0184fe8a74e2408684587f0ec21afdaf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fe3c756fa884922bf2454d9b45e066a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:47:19.352385 5519 tablet_service.cc:1467] Processing CreateTablet for tablet 0184fe8a74e2408684587f0ec21afdaf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fe3c756fa884922bf2454d9b45e066a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:47:19.352783 5519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0184fe8a74e2408684587f0ec21afdaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.352782 5659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0184fe8a74e2408684587f0ec21afdaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.355831 5894 tablet_bootstrap.cc:492] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12: Bootstrap starting.
I20260927 18:47:19.355876 5799 tablet_service.cc:1467] Processing CreateTablet for tablet 0184fe8a74e2408684587f0ec21afdaf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fe3c756fa884922bf2454d9b45e066a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:47:19.356220 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0184fe8a74e2408684587f0ec21afdaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.356885 5894 tablet_bootstrap.cc:654] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.359081 5895 tablet_bootstrap.cc:492] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: Bootstrap starting.
I20260927 18:47:19.359474 5893 tablet_bootstrap.cc:492] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4: Bootstrap starting.
I20260927 18:47:19.360113 5895 tablet_bootstrap.cc:654] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.360391 5894 tablet_bootstrap.cc:492] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12: No bootstrap required, opened a new log
I20260927 18:47:19.360477 5894 ts_tablet_manager.cc:1397] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 18:47:19.360500 5893 tablet_bootstrap.cc:654] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.360978 5894 raft_consensus.cc:348] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.361102 5894 raft_consensus.cc:374] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.361145 5894 raft_consensus.cc:729] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65aab0f618194c6e8d4cb4a89a165b12, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.361274 5894 consensus_queue.cc:260] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.361584 5894 ts_tablet_manager.cc:1428] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 18:47:19.361747 5895 tablet_bootstrap.cc:492] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: No bootstrap required, opened a new log
I20260927 18:47:19.361840 5895 ts_tablet_manager.cc:1397] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:19.362160 5893 tablet_bootstrap.cc:492] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4: No bootstrap required, opened a new log
I20260927 18:47:19.362265 5893 ts_tablet_manager.cc:1397] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:19.362308 5895 raft_consensus.cc:348] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.362442 5895 raft_consensus.cc:374] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.362483 5895 raft_consensus.cc:729] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1978810c554fc09c4f9682638a0263, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.362608 5895 consensus_queue.cc:260] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.362872 5895 ts_tablet_manager.cc:1428] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:19.362811 5893 raft_consensus.cc:348] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.362938 5893 raft_consensus.cc:374] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.362978 5893 raft_consensus.cc:729] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa67eb538a6f4b09b7a2c19c527780a4, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.363113 5893 consensus_queue.cc:260] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.363106 5901 tablet_replica.cc:439] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } }
I20260927 18:47:19.363247 5901 tablet_replica.cc:442] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:47:19.363395 5893 ts_tablet_manager.cc:1428] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:19.363610 5899 tablet_replica.cc:439] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } }
I20260927 18:47:19.363724 5899 tablet_replica.cc:442] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:47:19.363883 5903 tablet_replica.cc:439] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } }
I20260927 18:47:19.363982 5903 tablet_replica.cc:442] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:47:19.415148 5903 raft_consensus.cc:482] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:19.415298 5903 raft_consensus.cc:504] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.415865 5903 leader_election.cc:288] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709), 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253)
I20260927 18:47:19.416378 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184fe8a74e2408684587f0ec21afdaf" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65aab0f618194c6e8d4cb4a89a165b12" is_pre_election: true
I20260927 18:47:19.416385 5539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184fe8a74e2408684587f0ec21afdaf" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" is_pre_election: true
I20260927 18:47:19.416556 5539 raft_consensus.cc:2454] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 0.
I20260927 18:47:19.416510 5679 raft_consensus.cc:2454] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 0.
I20260927 18:47:19.416913 5750 leader_election.cc:302] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1978810c554fc09c4f9682638a0263, aa67eb538a6f4b09b7a2c19c527780a4; no voters:
I20260927 18:47:19.417138 5903 raft_consensus.cc:2781] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:19.417229 5903 raft_consensus.cc:482] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:19.417308 5903 raft_consensus.cc:3037] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.418480 5903 raft_consensus.cc:504] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.418897 5903 leader_election.cc:288] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 election: Requested vote from peers aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709), 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253)
I20260927 18:47:19.419234 5539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184fe8a74e2408684587f0ec21afdaf" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67eb538a6f4b09b7a2c19c527780a4"
I20260927 18:47:19.419360 5539 raft_consensus.cc:3037] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.420578 5539 raft_consensus.cc:2454] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 1.
I20260927 18:47:19.420858 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184fe8a74e2408684587f0ec21afdaf" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65aab0f618194c6e8d4cb4a89a165b12"
I20260927 18:47:19.421001 5679 raft_consensus.cc:3037] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.421010 5750 leader_election.cc:302] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1978810c554fc09c4f9682638a0263, aa67eb538a6f4b09b7a2c19c527780a4; no voters:
I20260927 18:47:19.421654 5903 raft_consensus.cc:2781] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:19.421762 5903 raft_consensus.cc:686] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [term 1 LEADER]: Becoming Leader. State: Replica: 6c1978810c554fc09c4f9682638a0263, State: Running, Role: LEADER
I20260927 18:47:19.422194 5679 raft_consensus.cc:2454] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 1.
I20260927 18:47:19.422147 5903 consensus_queue.cc:237] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.423314 5900 tablet_replica.cc:439] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: TxnStatusTablet state changed. Reason: New leader 6c1978810c554fc09c4f9682638a0263. Latest consensus state: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } }
I20260927 18:47:19.423445 5900 tablet_replica.cc:442] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:47:19.423738 5932 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:47:19.423920 5932 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:47:19.424047 5382 catalog_manager.cc:5495] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c1978810c554fc09c4f9682638a0263 (127.29.159.67). New cstate: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } health_report { overall_health: HEALTHY } } }
I20260927 18:47:19.447495 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.444111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.29.159.67:37917)
0927 18:47:19.444235 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.444257 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.444271 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:47:19.444456 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.444461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.444516 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.444596 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:47:19.444881 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.445393 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.446258 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.446482 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.446593 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.446947 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:47:19.447055 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:47:19.447133 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.447260 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.447317 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:47:19.447495 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.444013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:37917 (local address 127.0.0.1:58586)
0927 18:47:19.444184 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:19.444200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:19.444325 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:19.444629 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:19.444632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:19.444640 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:47:19.444771 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.444776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.445505 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.445510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.446124 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.446133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.446591 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.446595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.446700 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.446751 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:47:19.447150 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:47:19.447166 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 18:47:19.447233 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 18:47:19.447320 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 18:47:19.450372 5679 raft_consensus.cc:1261] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Refusing update from remote peer 6c1978810c554fc09c4f9682638a0263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:19.450414 5539 raft_consensus.cc:1261] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c1978810c554fc09c4f9682638a0263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:19.450932 5903 consensus_queue.cc:1035] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:19.451086 5900 consensus_queue.cc:1035] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:19.452512 5901 tablet_replica.cc:439] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12: TxnStatusTablet state changed. Reason: New leader 6c1978810c554fc09c4f9682638a0263. Latest consensus state: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } }
I20260927 18:47:19.452634 5901 tablet_replica.cc:442] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:47:19.455094 5899 tablet_replica.cc:439] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4: TxnStatusTablet state changed. Reason: New leader 6c1978810c554fc09c4f9682638a0263. Latest consensus state: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } }
I20260927 18:47:19.455227 5899 tablet_replica.cc:442] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:47:19.455499 5900 tablet_replica.cc:439] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } }
I20260927 18:47:19.455614 5900 tablet_replica.cc:442] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:47:19.455946 5900 tablet_replica.cc:439] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } }
I20260927 18:47:19.456054 5900 tablet_replica.cc:442] T 0184fe8a74e2408684587f0ec21afdaf P 6c1978810c554fc09c4f9682638a0263: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:47:19.456980 5899 tablet_replica.cc:439] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } }
I20260927 18:47:19.457087 5899 tablet_replica.cc:442] T 0184fe8a74e2408684587f0ec21afdaf P aa67eb538a6f4b09b7a2c19c527780a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:47:19.457540 5901 tablet_replica.cc:439] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } } }
I20260927 18:47:19.457651 5901 tablet_replica.cc:442] T 0184fe8a74e2408684587f0ec21afdaf P 65aab0f618194c6e8d4cb4a89a165b12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:47:19.457660 5907 mvcc.cc:205] Tried to move back new op lower bound from 7334030702384414720 to 7334030702274555904. Current Snapshot: MvccSnapshot[applied={T|T < 7334030702384414720 or (T in {7334030702384414720})}]
I20260927 18:47:19.472535 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.469186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:37917 (local address 127.29.159.67:58639)
0927 18:47:19.469362 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:19.469379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:19.469508 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:19.469757 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:19.469760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:19.469766 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:47:19.469929 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.469935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.470650 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.470653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.471251 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.471259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.471786 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.471790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.471896 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.471936 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:47:19.472229 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:47:19.472254 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:47:19.472318 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 18:47:19.472402 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":23}
I20260927 18:47:19.472535 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.469264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:58639 (local address 127.29.159.67:37917)
0927 18:47:19.469360 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.469365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.469449 (+ 84us) server_negotiation.cc:407] Connection header received
0927 18:47:19.469604 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.469608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.469657 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.469756 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:47:19.470029 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.470532 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.471367 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.471669 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.471788 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.472072 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:47:19.472148 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:47:19.472228 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.472339 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.472377 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":46}
I20260927 18:47:19.479599 5539 raft_consensus.cc:1261] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c1978810c554fc09c4f9682638a0263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:19.480000 5679 raft_consensus.cc:1261] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Refusing update from remote peer 6c1978810c554fc09c4f9682638a0263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:19.480422 5903 consensus_queue.cc:1035] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:19.480607 5936 consensus_queue.cc:1035] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:47:19.482792 5866 tablet.cc:2369] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:47:19.483682 5906 mvcc.cc:205] Tried to move back new op lower bound from 7334030702505082880 to 7334030701735272448. Current Snapshot: MvccSnapshot[applied={T|T < 7334030702505082880}]
I20260927 18:47:19.516074 5381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:19.516251 5381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:19.518261 5381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444: Sending DeleteTablet for 3 replicas of tablet a7f9268194fd4afbb1dba818be366900
I20260927 18:47:19.518829 5659 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f9268194fd4afbb1dba818be366900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:40298
I20260927 18:47:19.519037 5519 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f9268194fd4afbb1dba818be366900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:48968
I20260927 18:47:19.519232 5916 tablet_replica.cc:330] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12: stopping tablet replica
I20260927 18:47:19.519356 5916 raft_consensus.cc:2229] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:19.519490 5916 raft_consensus.cc:2258] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.519668 5917 tablet_replica.cc:330] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4: stopping tablet replica
I20260927 18:47:19.519644 5799 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f9268194fd4afbb1dba818be366900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:58582
I20260927 18:47:19.519785 5917 raft_consensus.cc:2229] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:19.519809 5915 tablet_replica.cc:330] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263: stopping tablet replica
I20260927 18:47:19.519912 5917 raft_consensus.cc:2258] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.519904 5915 raft_consensus.cc:2229] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:47:19.520112 5915 raft_consensus.cc:2258] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.520709 5916 ts_tablet_manager.cc:1905] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.520712 5917 ts_tablet_manager.cc:1905] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.520810 5915 ts_tablet_manager.cc:1905] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.523568 5916 ts_tablet_manager.cc:1918] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:47:19.523568 5915 ts_tablet_manager.cc:1918] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:47:19.523655 5915 log.cc:1196] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/a7f9268194fd4afbb1dba818be366900
I20260927 18:47:19.523655 5916 log.cc:1196] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/a7f9268194fd4afbb1dba818be366900
I20260927 18:47:19.523938 5916 ts_tablet_manager.cc:1939] T a7f9268194fd4afbb1dba818be366900 P 65aab0f618194c6e8d4cb4a89a165b12: Deleting consensus metadata
I20260927 18:47:19.523938 5915 ts_tablet_manager.cc:1939] T a7f9268194fd4afbb1dba818be366900 P 6c1978810c554fc09c4f9682638a0263: Deleting consensus metadata
I20260927 18:47:19.524082 5917 ts_tablet_manager.cc:1918] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:47:19.524169 5917 log.cc:1196] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/wal/wals/a7f9268194fd4afbb1dba818be366900
I20260927 18:47:19.524441 5917 ts_tablet_manager.cc:1939] T a7f9268194fd4afbb1dba818be366900 P aa67eb538a6f4b09b7a2c19c527780a4: Deleting consensus metadata
I20260927 18:47:19.524744 5369 catalog_manager.cc:4841] TS 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253): tablet a7f9268194fd4afbb1dba818be366900 (table test-table [id=5bdf2b6ea56943e9ad4878a292b183ae]) successfully deleted
I20260927 18:47:19.524752 5366 catalog_manager.cc:4841] TS 6c1978810c554fc09c4f9682638a0263 (127.29.159.67:37917): tablet a7f9268194fd4afbb1dba818be366900 (table test-table [id=5bdf2b6ea56943e9ad4878a292b183ae]) successfully deleted
I20260927 18:47:19.525758 5366 catalog_manager.cc:4841] TS aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709): tablet a7f9268194fd4afbb1dba818be366900 (table test-table [id=5bdf2b6ea56943e9ad4878a292b183ae]) successfully deleted
Sep 27 18:47:19 dist-test-slave-8tg2 krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:47:19 dist-test-slave-8tg2 krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:19 dist-test-slave-8tg2 krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:47:19.556502 5431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:19.549174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.29.159.126:38113)
0927 18:47:19.549356 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.549361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.549487 (+ 126us) server_negotiation.cc:407] Connection header received
0927 18:47:19.549579 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.549583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.549645 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.549745 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:19.550431 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.551340 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.552352 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.552769 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.552875 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.556361 (+ 3486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45428: BlockingRecv error: recv got EOF from 127.0.0.1:45428 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 27 18:47:19 dist-test-slave-8tg2 krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM
I20260927 18:47:19.567327 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.557992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.29.159.126:38113)
0927 18:47:19.558247 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.558252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.558508 (+ 256us) server_negotiation.cc:407] Connection header received
0927 18:47:19.558625 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.558628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.558668 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.558735 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:19.559509 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.560000 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.561227 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.561433 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.561508 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.564240 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:19.564244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:19.564247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:19.564271 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:19.565710 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:19.566136 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:19.566142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:19.566143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:19.566223 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:19.566497 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:19.566502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:19.566503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:19.566677 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:19.566822 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.567050 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.567158 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260927 18:47:19.570080 5381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:47:19.570562 5381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:47:19.575295 5659 tablet_service.cc:1467] Processing CreateTablet for tablet 27760cddf1094db88793fac6c64f659f (DEFAULT_TABLE table=test-table [id=776cdbc2847a4f65bb64ddeab7221e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.575273 5519 tablet_service.cc:1467] Processing CreateTablet for tablet 27760cddf1094db88793fac6c64f659f (DEFAULT_TABLE table=test-table [id=776cdbc2847a4f65bb64ddeab7221e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.575603 5659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27760cddf1094db88793fac6c64f659f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.575603 5519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27760cddf1094db88793fac6c64f659f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.575557 5799 tablet_service.cc:1467] Processing CreateTablet for tablet 27760cddf1094db88793fac6c64f659f (DEFAULT_TABLE table=test-table [id=776cdbc2847a4f65bb64ddeab7221e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.575899 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27760cddf1094db88793fac6c64f659f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.577963 5894 tablet_bootstrap.cc:492] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12: Bootstrap starting.
I20260927 18:47:19.577963 5893 tablet_bootstrap.cc:492] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4: Bootstrap starting.
I20260927 18:47:19.578485 5895 tablet_bootstrap.cc:492] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263: Bootstrap starting.
I20260927 18:47:19.578841 5894 tablet_bootstrap.cc:654] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.578841 5893 tablet_bootstrap.cc:654] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.579353 5895 tablet_bootstrap.cc:654] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.580551 5894 tablet_bootstrap.cc:492] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12: No bootstrap required, opened a new log
I20260927 18:47:19.580572 5893 tablet_bootstrap.cc:492] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4: No bootstrap required, opened a new log
I20260927 18:47:19.580636 5893 ts_tablet_manager.cc:1397] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:19.580636 5894 ts_tablet_manager.cc:1397] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:19.581064 5895 tablet_bootstrap.cc:492] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263: No bootstrap required, opened a new log
I20260927 18:47:19.581151 5895 ts_tablet_manager.cc:1397] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 18:47:19.581120 5894 raft_consensus.cc:348] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.581225 5894 raft_consensus.cc:374] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.581260 5894 raft_consensus.cc:729] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65aab0f618194c6e8d4cb4a89a165b12, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.581310 5893 raft_consensus.cc:348] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.581405 5893 raft_consensus.cc:374] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.581385 5894 consensus_queue.cc:260] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.581436 5893 raft_consensus.cc:729] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa67eb538a6f4b09b7a2c19c527780a4, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.581550 5893 consensus_queue.cc:260] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.581650 5894 ts_tablet_manager.cc:1428] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:19.581557 5895 raft_consensus.cc:348] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.581698 5895 raft_consensus.cc:374] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.581739 5895 raft_consensus.cc:729] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1978810c554fc09c4f9682638a0263, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.581787 5893 ts_tablet_manager.cc:1428] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:19.581867 5895 consensus_queue.cc:260] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.582139 5895 ts_tablet_manager.cc:1428] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 18:47:19.742883 5936 raft_consensus.cc:482] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:19.743032 5936 raft_consensus.cc:504] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.743505 5936 leader_election.cc:288] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709), 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253)
I20260927 18:47:19.743960 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27760cddf1094db88793fac6c64f659f" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65aab0f618194c6e8d4cb4a89a165b12" is_pre_election: true
I20260927 18:47:19.743963 5538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27760cddf1094db88793fac6c64f659f" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" is_pre_election: true
I20260927 18:47:19.744092 5679 raft_consensus.cc:2454] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 0.
I20260927 18:47:19.744134 5538 raft_consensus.cc:2454] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 0.
I20260927 18:47:19.744422 5753 leader_election.cc:302] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65aab0f618194c6e8d4cb4a89a165b12, 6c1978810c554fc09c4f9682638a0263; no voters:
I20260927 18:47:19.744617 5936 raft_consensus.cc:2781] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:19.744695 5936 raft_consensus.cc:482] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:19.744760 5936 raft_consensus.cc:3037] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.745786 5936 raft_consensus.cc:504] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.746177 5936 leader_election.cc:288] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 election: Requested vote from peers aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709), 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253)
I20260927 18:47:19.746548 5538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27760cddf1094db88793fac6c64f659f" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67eb538a6f4b09b7a2c19c527780a4"
I20260927 18:47:19.746681 5538 raft_consensus.cc:3037] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.746662 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27760cddf1094db88793fac6c64f659f" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65aab0f618194c6e8d4cb4a89a165b12"
I20260927 18:47:19.746860 5679 raft_consensus.cc:3037] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.747856 5538 raft_consensus.cc:2454] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 1.
I20260927 18:47:19.747979 5679 raft_consensus.cc:2454] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 1.
I20260927 18:47:19.748245 5750 leader_election.cc:302] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1978810c554fc09c4f9682638a0263, aa67eb538a6f4b09b7a2c19c527780a4; no voters:
I20260927 18:47:19.748409 5936 raft_consensus.cc:2781] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:19.748495 5936 raft_consensus.cc:686] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 1 LEADER]: Becoming Leader. State: Replica: 6c1978810c554fc09c4f9682638a0263, State: Running, Role: LEADER
I20260927 18:47:19.748638 5936 consensus_queue.cc:237] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.752689 5382 catalog_manager.cc:5495] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c1978810c554fc09c4f9682638a0263 (127.29.159.67). New cstate: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } health_report { overall_health: HEALTHY } } }
I20260927 18:47:19.778429 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.775417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.29.159.67:37917)
0927 18:47:19.775582 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.775586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.775599 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:47:19.775697 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.775699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.775741 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.775828 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:47:19.776045 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.776560 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.777449 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.777656 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.777735 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.778030 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:47:19.778112 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:47:19.778172 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.778271 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.778315 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:47:19.780550 5679 raft_consensus.cc:1261] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Refusing update from remote peer 6c1978810c554fc09c4f9682638a0263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:19.780563 5538 raft_consensus.cc:1261] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c1978810c554fc09c4f9682638a0263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:19.781018 5936 consensus_queue.cc:1035] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:19.781231 5900 consensus_queue.cc:1035] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:19.788671 5381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:19.788789 5381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:19.790714 5381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444: Sending DeleteTablet for 3 replicas of tablet 27760cddf1094db88793fac6c64f659f
I20260927 18:47:19.791225 5659 tablet_service.cc:1514] Processing DeleteTablet for tablet 27760cddf1094db88793fac6c64f659f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:40298
I20260927 18:47:19.791271 5519 tablet_service.cc:1514] Processing DeleteTablet for tablet 27760cddf1094db88793fac6c64f659f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:48968
I20260927 18:47:19.791460 5916 tablet_replica.cc:330] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12: stopping tablet replica
I20260927 18:47:19.791476 5917 tablet_replica.cc:330] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4: stopping tablet replica
I20260927 18:47:19.791486 5799 tablet_service.cc:1514] Processing DeleteTablet for tablet 27760cddf1094db88793fac6c64f659f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:58582
I20260927 18:47:19.791597 5916 raft_consensus.cc:2229] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:19.791595 5917 raft_consensus.cc:2229] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:19.791635 5915 tablet_replica.cc:330] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263: stopping tablet replica
I20260927 18:47:19.791734 5916 raft_consensus.cc:2258] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.791728 5915 raft_consensus.cc:2229] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:47:19.791726 5917 raft_consensus.cc:2258] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.791911 5915 raft_consensus.cc:2258] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.792500 5916 ts_tablet_manager.cc:1905] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.792568 5915 ts_tablet_manager.cc:1905] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.794565 5916 ts_tablet_manager.cc:1918] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:19.794651 5916 log.cc:1196] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/27760cddf1094db88793fac6c64f659f
I20260927 18:47:19.794559 5381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 18:47:19.794657 5917 ts_tablet_manager.cc:1905] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.794901 5916 ts_tablet_manager.cc:1939] T 27760cddf1094db88793fac6c64f659f P 65aab0f618194c6e8d4cb4a89a165b12: Deleting consensus metadata
W20260927 18:47:19.795022 5381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:47:19.795609 5915 ts_tablet_manager.cc:1918] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:19.795609 5369 catalog_manager.cc:4841] TS 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253): tablet 27760cddf1094db88793fac6c64f659f (table test-table [id=776cdbc2847a4f65bb64ddeab7221e1e]) successfully deleted
I20260927 18:47:19.795692 5915 log.cc:1196] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/27760cddf1094db88793fac6c64f659f
I20260927 18:47:19.795907 5915 ts_tablet_manager.cc:1939] T 27760cddf1094db88793fac6c64f659f P 6c1978810c554fc09c4f9682638a0263: Deleting consensus metadata
I20260927 18:47:19.796461 5366 catalog_manager.cc:4841] TS 6c1978810c554fc09c4f9682638a0263 (127.29.159.67:37917): tablet 27760cddf1094db88793fac6c64f659f (table test-table [id=776cdbc2847a4f65bb64ddeab7221e1e]) successfully deleted
I20260927 18:47:19.797031 5917 ts_tablet_manager.cc:1918] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:19.797114 5917 log.cc:1196] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-0/wal/wals/27760cddf1094db88793fac6c64f659f
I20260927 18:47:19.797372 5917 ts_tablet_manager.cc:1939] T 27760cddf1094db88793fac6c64f659f P aa67eb538a6f4b09b7a2c19c527780a4: Deleting consensus metadata
I20260927 18:47:19.797988 5366 catalog_manager.cc:4841] TS aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709): tablet 27760cddf1094db88793fac6c64f659f (table test-table [id=776cdbc2847a4f65bb64ddeab7221e1e]) successfully deleted
I20260927 18:47:19.800420 5519 tablet_service.cc:1467] Processing CreateTablet for tablet 574f2928312f46c99aa2df96c163148f (DEFAULT_TABLE table=test-table [id=afc2f1cf85a145bba447fd7a6a8f13c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.800598 5799 tablet_service.cc:1467] Processing CreateTablet for tablet 574f2928312f46c99aa2df96c163148f (DEFAULT_TABLE table=test-table [id=afc2f1cf85a145bba447fd7a6a8f13c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.800683 5519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574f2928312f46c99aa2df96c163148f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.800870 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574f2928312f46c99aa2df96c163148f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.802398 5893 tablet_bootstrap.cc:492] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4: Bootstrap starting.
I20260927 18:47:19.802764 5895 tablet_bootstrap.cc:492] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263: Bootstrap starting.
I20260927 18:47:19.802736 5659 tablet_service.cc:1467] Processing CreateTablet for tablet 574f2928312f46c99aa2df96c163148f (DEFAULT_TABLE table=test-table [id=afc2f1cf85a145bba447fd7a6a8f13c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:19.802987 5659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574f2928312f46c99aa2df96c163148f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:19.803359 5893 tablet_bootstrap.cc:654] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.803608 5895 tablet_bootstrap.cc:654] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.804749 5893 tablet_bootstrap.cc:492] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4: No bootstrap required, opened a new log
I20260927 18:47:19.804837 5893 ts_tablet_manager.cc:1397] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:19.805274 5893 raft_consensus.cc:348] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.805393 5893 raft_consensus.cc:374] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.805435 5893 raft_consensus.cc:729] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa67eb538a6f4b09b7a2c19c527780a4, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.805562 5893 consensus_queue.cc:260] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.805797 5894 tablet_bootstrap.cc:492] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12: Bootstrap starting.
I20260927 18:47:19.806407 5895 tablet_bootstrap.cc:492] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263: No bootstrap required, opened a new log
I20260927 18:47:19.806509 5895 ts_tablet_manager.cc:1397] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 18:47:19.806749 5894 tablet_bootstrap.cc:654] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:19.806913 5895 raft_consensus.cc:348] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.807022 5895 raft_consensus.cc:374] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.807065 5895 raft_consensus.cc:729] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1978810c554fc09c4f9682638a0263, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.807192 5895 consensus_queue.cc:260] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.807515 5895 ts_tablet_manager.cc:1428] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 18:47:19.807914 5893 ts_tablet_manager.cc:1428] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 18:47:19.808068 5894 tablet_bootstrap.cc:492] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12: No bootstrap required, opened a new log
I20260927 18:47:19.808141 5894 ts_tablet_manager.cc:1397] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:19.808543 5894 raft_consensus.cc:348] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.808621 5894 raft_consensus.cc:374] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:19.808641 5894 raft_consensus.cc:729] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65aab0f618194c6e8d4cb4a89a165b12, State: Initialized, Role: FOLLOWER
I20260927 18:47:19.808777 5894 consensus_queue.cc:260] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.809043 5894 ts_tablet_manager.cc:1428] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 18:47:19.901778 5900 raft_consensus.cc:482] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:19.901947 5900 raft_consensus.cc:504] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.902184 5899 raft_consensus.cc:482] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:19.902297 5899 raft_consensus.cc:504] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.902504 5900 leader_election.cc:288] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709), 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253)
I20260927 18:47:19.902933 5538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574f2928312f46c99aa2df96c163148f" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" is_pre_election: true
I20260927 18:47:19.902967 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574f2928312f46c99aa2df96c163148f" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65aab0f618194c6e8d4cb4a89a165b12" is_pre_election: true
I20260927 18:47:19.903077 5538 raft_consensus.cc:2454] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 0.
I20260927 18:47:19.903098 5679 raft_consensus.cc:2454] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 0.
I20260927 18:47:19.903471 5750 leader_election.cc:302] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1978810c554fc09c4f9682638a0263, aa67eb538a6f4b09b7a2c19c527780a4; no voters:
I20260927 18:47:19.903694 5900 raft_consensus.cc:2781] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:19.903756 5900 raft_consensus.cc:482] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:19.903797 5900 raft_consensus.cc:3037] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.903892 5899 leader_election.cc:288] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253), 6c1978810c554fc09c4f9682638a0263 (127.29.159.67:37917)
I20260927 18:47:19.905158 5900 raft_consensus.cc:504] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.905531 5900 leader_election.cc:288] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 election: Requested vote from peers aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709), 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253)
I20260927 18:47:19.905812 5538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574f2928312f46c99aa2df96c163148f" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67eb538a6f4b09b7a2c19c527780a4"
I20260927 18:47:19.905895 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574f2928312f46c99aa2df96c163148f" candidate_uuid: "6c1978810c554fc09c4f9682638a0263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65aab0f618194c6e8d4cb4a89a165b12"
I20260927 18:47:19.905932 5538 raft_consensus.cc:3037] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.906008 5679 raft_consensus.cc:3037] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:19.907071 5538 raft_consensus.cc:2454] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 1.
I20260927 18:47:19.907073 5679 raft_consensus.cc:2454] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1978810c554fc09c4f9682638a0263 in term 1.
I20260927 18:47:19.907390 5750 leader_election.cc:302] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1978810c554fc09c4f9682638a0263, aa67eb538a6f4b09b7a2c19c527780a4; no voters:
I20260927 18:47:19.907616 5900 raft_consensus.cc:2781] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:19.907776 5900 raft_consensus.cc:686] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 1 LEADER]: Becoming Leader. State: Replica: 6c1978810c554fc09c4f9682638a0263, State: Running, Role: LEADER
I20260927 18:47:19.907922 5900 consensus_queue.cc:237] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } }
I20260927 18:47:19.909253 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.904220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:32949 (local address 127.29.159.67:37917)
0927 18:47:19.905000 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.905003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.905013 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:47:19.905047 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.905049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.905118 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.905198 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:19.905558 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.906309 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.907960 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.908534 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.908627 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.908678 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.909048 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.909123 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702}
I20260927 18:47:19.909641 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.904143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:37917 (local address 127.29.159.65:32949)
0927 18:47:19.904669 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:19.904686 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:19.904826 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:19.905304 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:19.905308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:19.905323 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:19.905439 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.905445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.906476 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.906479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.907826 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.907836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.908773 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.908776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.908888 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.908913 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:47:19.909373 (+ 460us) client_negotiation.cc:240] Negotiation successful
0927 18:47:19.909462 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":71,"thread_start_us":74,"threads_started":1}
I20260927 18:47:19.909675 5382 catalog_manager.cc:5495] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c1978810c554fc09c4f9682638a0263 (127.29.159.67). New cstate: current_term: 1 leader_uuid: "6c1978810c554fc09c4f9682638a0263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c1978810c554fc09c4f9682638a0263" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 37917 } health_report { overall_health: HEALTHY } } }
I20260927 18:47:19.910046 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.904145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:33253 (local address 127.29.159.65:59347)
0927 18:47:19.904746 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:19.904759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:19.904832 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:19.905125 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:19.905128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:19.905150 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:19.905296 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.905302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.906485 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.906488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.908352 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:19.908362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.909616 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.909619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:19.909738 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.909771 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:47:19.909825 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 18:47:19.909907 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":678,"thread_start_us":82,"threads_started":1}
I20260927 18:47:19.910157 5819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574f2928312f46c99aa2df96c163148f" candidate_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c1978810c554fc09c4f9682638a0263" is_pre_election: true
I20260927 18:47:19.910601 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.904299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:59347 (local address 127.29.159.66:33253)
0927 18:47:19.904564 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:19.904569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:19.904837 (+ 268us) server_negotiation.cc:407] Connection header received
0927 18:47:19.904937 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:19.904940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:19.905004 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:19.905099 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:19.905428 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.906325 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.908873 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:19.909488 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:19.910296 (+ 808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:19.910367 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:19.910406 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 18:47:19.910475 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260927 18:47:19.910977 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574f2928312f46c99aa2df96c163148f" candidate_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65aab0f618194c6e8d4cb4a89a165b12" is_pre_election: true
I20260927 18:47:19.911162 5679 raft_consensus.cc:2379] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa67eb538a6f4b09b7a2c19c527780a4 in current term 1: Already voted for candidate 6c1978810c554fc09c4f9682638a0263 in this term.
I20260927 18:47:19.911542 5473 leader_election.cc:302] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa67eb538a6f4b09b7a2c19c527780a4; no voters: 65aab0f618194c6e8d4cb4a89a165b12, 6c1978810c554fc09c4f9682638a0263
I20260927 18:47:19.911821 5899 raft_consensus.cc:2726] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 18:47:19.938010 5679 raft_consensus.cc:1261] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Refusing update from remote peer 6c1978810c554fc09c4f9682638a0263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:19.938050 5538 raft_consensus.cc:1261] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c1978810c554fc09c4f9682638a0263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:19.938434 5900 consensus_queue.cc:1035] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65aab0f618194c6e8d4cb4a89a165b12" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:19.938619 5903 consensus_queue.cc:1035] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa67eb538a6f4b09b7a2c19c527780a4" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:19.970598 5382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:19.970765 5382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:19.972916 5382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58af39a3b8024e46a80ff3482c28f444: Sending DeleteTablet for 3 replicas of tablet 574f2928312f46c99aa2df96c163148f
I20260927 18:47:19.973558 5799 tablet_service.cc:1514] Processing DeleteTablet for tablet 574f2928312f46c99aa2df96c163148f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:58582
I20260927 18:47:19.973568 5659 tablet_service.cc:1514] Processing DeleteTablet for tablet 574f2928312f46c99aa2df96c163148f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:40298
I20260927 18:47:19.973672 5519 tablet_service.cc:1514] Processing DeleteTablet for tablet 574f2928312f46c99aa2df96c163148f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:19 UTC) from {username='oryx', principal='oryx/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:48968
I20260927 18:47:19.973744 5915 tablet_replica.cc:330] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263: stopping tablet replica
I20260927 18:47:19.973834 5916 tablet_replica.cc:330] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12: stopping tablet replica
I20260927 18:47:19.973850 5915 raft_consensus.cc:2229] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:47:19.973904 5917 tablet_replica.cc:330] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4: stopping tablet replica
I20260927 18:47:19.973941 5916 raft_consensus.cc:2229] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:19.974036 5916 raft_consensus.cc:2258] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.974018 5917 raft_consensus.cc:2229] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:19.974042 5915 raft_consensus.cc:2258] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.974140 5917 raft_consensus.cc:2258] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:19.974886 5915 ts_tablet_manager.cc:1905] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.974977 5917 ts_tablet_manager.cc:1905] T 574f2928312f46c99aa2df96c163148f P aa67eb538a6f4b09b7a2c19c527780a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.974994 5916 ts_tablet_manager.cc:1905] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:19.975242 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 5454
I20260927 18:47:19.977568 5916 ts_tablet_manager.cc:1918] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:47:19.977664 5916 log.cc:1196] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/574f2928312f46c99aa2df96c163148f
I20260927 18:47:19.977903 5916 ts_tablet_manager.cc:1939] T 574f2928312f46c99aa2df96c163148f P 65aab0f618194c6e8d4cb4a89a165b12: Deleting consensus metadata
I20260927 18:47:19.978106 5915 ts_tablet_manager.cc:1918] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:47:19.978204 5915 log.cc:1196] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/574f2928312f46c99aa2df96c163148f
I20260927 18:47:19.978461 5915 ts_tablet_manager.cc:1939] T 574f2928312f46c99aa2df96c163148f P 6c1978810c554fc09c4f9682638a0263: Deleting consensus metadata
I20260927 18:47:19.978835 5369 catalog_manager.cc:4841] TS 65aab0f618194c6e8d4cb4a89a165b12 (127.29.159.66:33253): tablet 574f2928312f46c99aa2df96c163148f (table test-table [id=afc2f1cf85a145bba447fd7a6a8f13c5]) successfully deleted
I20260927 18:47:19.979122 5366 catalog_manager.cc:4841] TS 6c1978810c554fc09c4f9682638a0263 (127.29.159.67:37917): tablet 574f2928312f46c99aa2df96c163148f (table test-table [id=afc2f1cf85a145bba447fd7a6a8f13c5]) successfully deleted
W20260927 18:47:19.983583 5366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:36709 (error 108)
I20260927 18:47:19.983688 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 5594
I20260927 18:47:19.986197 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:19.985832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:36709 (local address 127.0.0.1:48974)
0927 18:47:19.986036 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:19.986109 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
W20260927 18:47:19.986414 5366 catalog_manager.cc:4584] TS aa67eb538a6f4b09b7a2c19c527780a4 (127.29.159.65:36709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 574f2928312f46c99aa2df96c163148f: Network error: Client connection negotiation failed: client connection to 127.29.159.65:36709: connect: Connection refused (error 111)
I20260927 18:47:19.990855 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 5734
I20260927 18:47:19.997668 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 5350
2026-09-27T18:47:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3140 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:47:20 dist-test-slave-8tg2 krb5kdc[5974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:47:20 dist-test-slave-8tg2 krb5kdc[5974](info): set up 2 sockets
Sep 27 18:47:20 dist-test-slave-8tg2 krb5kdc[5974](info): commencing operation
krb5kdc: starting...
W20260927 18:47:21.052253 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:47:21 dist-test-slave-8tg2 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:47:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:47:21.179242 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44791
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:44791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:21.287955 5987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:21.288208 5987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:21.288287 5987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:21.292295 5987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:21.292349 5987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:21.292368 5987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:21.292385 5987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:21.292445 5987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:21.297566 5987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:44791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:21.298599 5987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:21.299973 5987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:21.308910 5995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:21.308924 5993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:21.308964 5987 server_base.cc:1034] running on GCE node
W20260927 18:47:21.308923 5992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:21.309666 5987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:21.310765 5987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:21.311964 5987 hybrid_clock.cc:648] HybridClock initialized: now 1790534841311901 us; error 77 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T18:47:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1317 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:47:21 dist-test-slave-8tg2 krb5kdc[6000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:47:21 dist-test-slave-8tg2 krb5kdc[6000](info): set up 2 sockets
Sep 27 18:47:21 dist-test-slave-8tg2 krb5kdc[6000](info): commencing operation
krb5kdc: starting...
W20260927 18:47:22.364182 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:47:22 dist-test-slave-8tg2 krb5kdc[6000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:47:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:47:22.487272 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:39873
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:22.598243 6013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:22.598512 6013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:22.598553 6013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:22.602491 6013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:22.602555 6013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:22.602576 6013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:22.602597 6013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:22.602613 6013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:22.607852 6013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:22.608850 6013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:22.609848 6013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:22.618139 6018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:22.618333 6021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:22.618162 6019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:22.618450 6013 server_base.cc:1034] running on GCE node
I20260927 18:47:22.619124 6013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:22.620325 6013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:22.621619 6013 hybrid_clock.cc:648] HybridClock initialized: now 1790534842621607 us; error 27 us; skew 500 ppm
Sep 27 18:47:22 dist-test-slave-8tg2 krb5kdc[6000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534842, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:22.624639 6013 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:47:22.625604 6013 webserver.cc:460] Webserver started at http://127.29.159.126:43381/ using document root <none> and password file <none>
I20260927 18:47:22.626183 6013 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:22.626295 6013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:22.626535 6013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:22.628641 6013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "5209703f3ee14be287083cfdf3516cf4"
format_stamp: "Formatted at 2026-09-27 18:47:22 on dist-test-slave-8tg2"
server_key: "16a801578e6d933a2fbec25d9d21ddae"
server_key_iv: "38178b3ad6d7d2f9f973bef02c9b8f81"
server_key_version: "encryptionkey@0"
I20260927 18:47:22.629202 6013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "5209703f3ee14be287083cfdf3516cf4"
format_stamp: "Formatted at 2026-09-27 18:47:22 on dist-test-slave-8tg2"
server_key: "16a801578e6d933a2fbec25d9d21ddae"
server_key_iv: "38178b3ad6d7d2f9f973bef02c9b8f81"
server_key_version: "encryptionkey@0"
I20260927 18:47:22.632882 6013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:47:22.635272 6028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:22.636564 6013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:22.636708 6013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "5209703f3ee14be287083cfdf3516cf4"
format_stamp: "Formatted at 2026-09-27 18:47:22 on dist-test-slave-8tg2"
server_key: "16a801578e6d933a2fbec25d9d21ddae"
server_key_iv: "38178b3ad6d7d2f9f973bef02c9b8f81"
server_key_version: "encryptionkey@0"
I20260927 18:47:22.636823 6013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:22.650408 6013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:22.653115 6013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:22.653317 6013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:22.660699 6013 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:39873
I20260927 18:47:22.660739 6080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:39873 every 8 connection(s)
I20260927 18:47:22.661626 6013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:22.663592 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 6013
I20260927 18:47:22.663724 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:22.664011 30333 external_mini_cluster.cc:1428] Setting key 3c822b7da447b9100594e877b70bf784
I20260927 18:47:22.664443 6081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:22.671164 6081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4: Bootstrap starting.
Sep 27 18:47:22 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:22.673743 6081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:22.674448 6081 log.cc:824] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:22.676651 6081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4: No bootstrap required, opened a new log
I20260927 18:47:22.678560 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:22.665383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.29.159.126:39873)
0927 18:47:22.665779 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:22.665795 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:22.665829 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:47:22.666459 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:22.666474 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:22.666786 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:22.667050 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:22.667961 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:22.668930 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:22.670144 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:22.670606 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:22.670724 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:22.673487 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:22.673492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:22.673505 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:22.673530 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:22.675537 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:22.675997 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:22.676002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:22.676006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:22.676118 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:22.676401 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:22.676403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:22.676404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:22.676822 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:22.676980 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:22.677274 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 18:47:22.677622 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":145,"threads_started":1}
I20260927 18:47:22.680177 6081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } }
I20260927 18:47:22.680446 6081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:22.680495 6081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5209703f3ee14be287083cfdf3516cf4, State: Initialized, Role: FOLLOWER
I20260927 18:47:22.680960 6081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } }
I20260927 18:47:22.681121 6081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:22.681171 6081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:22.681242 6081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:22.682118 6081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } }
I20260927 18:47:22.682415 6081 leader_election.cc:302] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5209703f3ee14be287083cfdf3516cf4; no voters:
I20260927 18:47:22.682664 6081 leader_election.cc:288] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:22.682809 6086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:22.683032 6086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 1 LEADER]: Becoming Leader. State: Replica: 5209703f3ee14be287083cfdf3516cf4, State: Running, Role: LEADER
I20260927 18:47:22.683359 6086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } }
I20260927 18:47:22.683998 6081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:22.684985 6087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5209703f3ee14be287083cfdf3516cf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } } }
I20260927 18:47:22.685103 6087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:22.685495 6095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:22.685438 6088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5209703f3ee14be287083cfdf3516cf4. Latest consensus state: current_term: 1 leader_uuid: "5209703f3ee14be287083cfdf3516cf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } } }
I20260927 18:47:22.685600 6088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:22.688623 6095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:22.694341 6095 catalog_manager.cc:1347] Generated new cluster ID: 54ddbb1dad064f2ab19ef17e990e397e
I20260927 18:47:22.694410 6095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:22.700821 6095 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:22.701810 6095 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:22.708104 6095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4: Generated new TSK 0
I20260927 18:47:22.708834 6095 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:47:22.742555 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39873
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:22.860219 6107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:22.860456 6107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:22.860523 6107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:22.864588 6107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:22.864640 6107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:22.864737 6107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:22.870023 6107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:22.871068 6107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:22.872171 6107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:22.881976 6112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:22.881970 6113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:22.882043 6107 server_base.cc:1034] running on GCE node
W20260927 18:47:22.881973 6115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:22.882586 6107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:22.883210 6107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:22.884397 6107 hybrid_clock.cc:648] HybridClock initialized: now 1790534842884382 us; error 28 us; skew 500 ppm
Sep 27 18:47:22 dist-test-slave-8tg2 krb5kdc[6000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534842, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:22.887391 6107 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:47:22.888304 6107 webserver.cc:460] Webserver started at http://127.29.159.65:37277/ using document root <none> and password file <none>
I20260927 18:47:22.888878 6107 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:22.888939 6107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:22.889197 6107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:22.891179 6107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "fbb2389032d64e87ac8862d502210ec6"
format_stamp: "Formatted at 2026-09-27 18:47:22 on dist-test-slave-8tg2"
server_key: "78c98aadd56d84ac0c2134dfe5e28d1e"
server_key_iv: "707ee86d1295262567ecd1b170e81120"
server_key_version: "encryptionkey@0"
I20260927 18:47:22.891759 6107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "fbb2389032d64e87ac8862d502210ec6"
format_stamp: "Formatted at 2026-09-27 18:47:22 on dist-test-slave-8tg2"
server_key: "78c98aadd56d84ac0c2134dfe5e28d1e"
server_key_iv: "707ee86d1295262567ecd1b170e81120"
server_key_version: "encryptionkey@0"
I20260927 18:47:22.895538 6107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:47:22.897994 6122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:22.899250 6107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:22.899374 6107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "fbb2389032d64e87ac8862d502210ec6"
format_stamp: "Formatted at 2026-09-27 18:47:22 on dist-test-slave-8tg2"
server_key: "78c98aadd56d84ac0c2134dfe5e28d1e"
server_key_iv: "707ee86d1295262567ecd1b170e81120"
server_key_version: "encryptionkey@0"
I20260927 18:47:22.899507 6107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:22.919350 6107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:22.922300 6107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:22.922501 6107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:22.923079 6107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:22.923907 6107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:22.923954 6107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:22.924018 6107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:22.924050 6107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:22.933087 6107 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:38787
I20260927 18:47:22.933108 6235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:38787 every 8 connection(s)
I20260927 18:47:22.933983 6107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:22.939532 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 6107
I20260927 18:47:22.939688 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:47:22.940063 30333 external_mini_cluster.cc:1428] Setting key 52e3a087ff47ae86260b1ef5cfc8a734
Sep 27 18:47:22 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534842, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:22.950446 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:22.935798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:50733 (local address 127.29.159.126:39873)
0927 18:47:22.935948 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:22.935953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:22.936881 (+ 928us) server_negotiation.cc:407] Connection header received
0927 18:47:22.938442 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:22.938445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:22.938493 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:22.938578 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:22.940206 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:22.941079 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:22.942554 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:22.942971 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:22.943106 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:22.946037 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:22.946045 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:22.946051 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:22.946090 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:22.948124 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:22.948647 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:22.948655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:22.948658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:22.948764 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:22.949145 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:22.949149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:22.949151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:22.949430 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:22.949552 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:22.950054 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 18:47:22.950210 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:47:22.951210 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:22.936261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:50733)
0927 18:47:22.936720 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:22.936765 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:22.938206 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:22.938858 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:22.938867 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:22.939293 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:22.940033 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:22.940051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:22.941239 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:22.941243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:22.942422 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:22.942431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:22.943140 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:22.943145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:22.943356 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:22.944117 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:22.944137 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:22.945810 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:22.948252 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:22.948257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:22.948270 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:22.948526 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:22.948890 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:22.948893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:22.948894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:22.949036 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:22.949547 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:22.949552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:22.949789 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 18:47:22.949951 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 18:47:22.950211 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260927 18:47:22.952545 6236 heartbeater.cc:347] Connected to a master server at 127.29.159.126:39873
I20260927 18:47:22.952787 6236 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:22.953326 6236 heartbeater.cc:511] Master 127.29.159.126:39873 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
I20260927 18:47:22.955183 6045 ts_manager.cc:194] Registered new tserver with Master: fbb2389032d64e87ac8862d502210ec6 (127.29.159.65:38787)
I20260927 18:47:22.956370 6045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:50733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:47:22.970623 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39873
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:23.081928 6241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:23.082203 6241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:23.082247 6241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:23.086650 6241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:23.086717 6241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:23.086819 6241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:23.092461 6241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:23.093669 6241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:23.094764 6241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:23.105111 6246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:23.105230 6247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:23.105129 6249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:23.105376 6241 server_base.cc:1034] running on GCE node
I20260927 18:47:23.105892 6241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:23.106660 6241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:23.107867 6241 hybrid_clock.cc:648] HybridClock initialized: now 1790534843107846 us; error 34 us; skew 500 ppm
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534843, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:23.110845 6241 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:47:23.111824 6241 webserver.cc:460] Webserver started at http://127.29.159.66:41827/ using document root <none> and password file <none>
I20260927 18:47:23.112377 6241 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:23.112437 6241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:23.112604 6241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:23.114614 6241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "bb150c4450a746cf9bfaef07a05daa7a"
format_stamp: "Formatted at 2026-09-27 18:47:23 on dist-test-slave-8tg2"
server_key: "7ab22a5fd449d40117a7fa3404a169be"
server_key_iv: "1b7c38e6553dc8279421c16fe7c4220d"
server_key_version: "encryptionkey@0"
I20260927 18:47:23.115134 6241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "bb150c4450a746cf9bfaef07a05daa7a"
format_stamp: "Formatted at 2026-09-27 18:47:23 on dist-test-slave-8tg2"
server_key: "7ab22a5fd449d40117a7fa3404a169be"
server_key_iv: "1b7c38e6553dc8279421c16fe7c4220d"
server_key_version: "encryptionkey@0"
I20260927 18:47:23.118953 6241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:47:23.121459 6256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:23.122704 6241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:23.122816 6241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "bb150c4450a746cf9bfaef07a05daa7a"
format_stamp: "Formatted at 2026-09-27 18:47:23 on dist-test-slave-8tg2"
server_key: "7ab22a5fd449d40117a7fa3404a169be"
server_key_iv: "1b7c38e6553dc8279421c16fe7c4220d"
server_key_version: "encryptionkey@0"
I20260927 18:47:23.122946 6241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:23.141744 6241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:23.145170 6241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:23.145382 6241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:23.146030 6241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:23.146895 6241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:23.146940 6241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:23.147017 6241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:23.147056 6241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:23.155988 6241 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:34303
I20260927 18:47:23.156013 6369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:34303 every 8 connection(s)
I20260927 18:47:23.156869 6241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:23.157212 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 6241
I20260927 18:47:23.157356 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:23.157721 30333 external_mini_cluster.cc:1428] Setting key 50980075fe63fe2b3d8dd01e2e8b4394
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534843, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:47:23.180609 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:23.158673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:50949 (local address 127.29.159.126:39873)
0927 18:47:23.158798 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:23.158807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:23.159934 (+ 1127us) server_negotiation.cc:407] Connection header received
0927 18:47:23.161646 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:23.161649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:23.161693 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:23.161771 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:23.163372 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.163927 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.165780 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.165990 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.166069 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:23.168817 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:23.168820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:23.168823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:23.168846 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:23.178223 (+ 9377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:23.178819 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:23.178823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:23.178824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:23.178904 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:23.179276 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:23.179279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:23.179280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:23.179582 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:23.179678 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:23.180320 (+ 642us) server_negotiation.cc:299] Negotiation successful
0927 18:47:23.180467 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:47:23.181548 6372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:23.159229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:50949)
0927 18:47:23.159768 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:23.159820 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:23.161410 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:23.161961 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:23.161974 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:23.162495 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:23.163185 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:23.163207 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.164056 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.164062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:23.165647 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:23.165661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.166152 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.166158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:23.166388 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:23.166986 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:23.167009 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:23.168654 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:23.178361 (+ 9707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:23.178367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:23.178383 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:23.178668 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:23.178996 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:23.178998 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:23.179000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:23.179154 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:23.179688 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:23.179695 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:23.179997 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 18:47:23.180228 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 18:47:23.180511 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":193,"threads_started":1}
I20260927 18:47:23.182801 6370 heartbeater.cc:347] Connected to a master server at 127.29.159.126:39873
I20260927 18:47:23.183019 6370 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:23.183576 6370 heartbeater.cc:511] Master 127.29.159.126:39873 requested a full tablet report, sending...
I20260927 18:47:23.184772 6045 ts_manager.cc:194] Registered new tserver with Master: bb150c4450a746cf9bfaef07a05daa7a (127.29.159.66:34303)
I20260927 18:47:23.185480 6045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:50949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:47:23.186667 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39873
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:23.295611 6375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:23.295881 6375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:23.295923 6375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:23.300414 6375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:23.300491 6375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:23.300585 6375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:23.306198 6375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:23.307296 6375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:23.308377 6375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:23.317818 6381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:23.317910 6375 server_base.cc:1034] running on GCE node
W20260927 18:47:23.317834 6380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:23.317843 6383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:23.318559 6375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:23.319226 6375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:23.320393 6375 hybrid_clock.cc:648] HybridClock initialized: now 1790534843320380 us; error 29 us; skew 500 ppm
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534843, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:23.323180 6375 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:47:23.324110 6375 webserver.cc:460] Webserver started at http://127.29.159.67:43563/ using document root <none> and password file <none>
I20260927 18:47:23.324680 6375 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:23.324741 6375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:23.324949 6375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:23.327108 6375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "c7a699639e9544d185114fafa12d43e8"
format_stamp: "Formatted at 2026-09-27 18:47:23 on dist-test-slave-8tg2"
server_key: "c6ff971bff125d900cd972f9cc482a5d"
server_key_iv: "2a79ae288ef32e9004a14559bc0365fe"
server_key_version: "encryptionkey@0"
I20260927 18:47:23.327710 6375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "c7a699639e9544d185114fafa12d43e8"
format_stamp: "Formatted at 2026-09-27 18:47:23 on dist-test-slave-8tg2"
server_key: "c6ff971bff125d900cd972f9cc482a5d"
server_key_iv: "2a79ae288ef32e9004a14559bc0365fe"
server_key_version: "encryptionkey@0"
I20260927 18:47:23.331493 6375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:47:23.333905 6390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:23.335202 6375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:23.335345 6375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "c7a699639e9544d185114fafa12d43e8"
format_stamp: "Formatted at 2026-09-27 18:47:23 on dist-test-slave-8tg2"
server_key: "c6ff971bff125d900cd972f9cc482a5d"
server_key_iv: "2a79ae288ef32e9004a14559bc0365fe"
server_key_version: "encryptionkey@0"
I20260927 18:47:23.335521 6375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:23.351686 6375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:23.355073 6375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:23.355286 6375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:23.356012 6375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:23.356873 6375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:23.356928 6375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:23.356968 6375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:23.357028 6375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:23.365945 6375 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:36185
I20260927 18:47:23.365970 6503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:36185 every 8 connection(s)
I20260927 18:47:23.366822 6375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:23.372556 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 6375
I20260927 18:47:23.372694 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:23.372978 30333 external_mini_cluster.cc:1428] Setting key ecd5bd31d53877ba26f358d3e6620077
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534843, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:23.380789 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:23.368620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:56387 (local address 127.29.159.126:39873)
0927 18:47:23.368769 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:23.368772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:23.369540 (+ 768us) server_negotiation.cc:407] Connection header received
0927 18:47:23.370900 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:23.370903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:23.370960 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:23.371067 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:23.372308 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.373125 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.374710 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.374960 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.375059 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:23.377501 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:23.377504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:23.377507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:23.377529 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:23.378770 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:23.379231 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:23.379235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:23.379237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:23.379315 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:23.379898 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:23.379901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:23.379902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:23.380062 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:23.380139 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:23.380578 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 18:47:23.380666 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:47:23.381583 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:23.368934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:56387)
0927 18:47:23.369376 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:23.369418 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:23.370681 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:23.371248 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:23.371255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:23.371622 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:23.372129 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:23.372143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.373245 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.373248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:23.374594 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:23.374604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.375114 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.375118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:23.375301 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:23.375885 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:23.375900 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:23.377354 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:23.378888 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:23.378893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:23.378904 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:23.379144 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:23.379402 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:23.379405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:23.379406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:23.379812 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:23.380155 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:23.380161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:23.380376 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 18:47:23.380523 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 18:47:23.380773 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260927 18:47:23.382664 6504 heartbeater.cc:347] Connected to a master server at 127.29.159.126:39873
I20260927 18:47:23.382860 6504 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:23.383284 6504 heartbeater.cc:511] Master 127.29.159.126:39873 requested a full tablet report, sending...
I20260927 18:47:23.384248 6045 ts_manager.cc:194] Registered new tserver with Master: c7a699639e9544d185114fafa12d43e8 (127.29.159.67:36185)
I20260927 18:47:23.384902 6045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:56387
I20260927 18:47:23.386792 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab.
Entry for principal oryx/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab.
WARNING: no policy specified for oryx/127.29.159.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab.
Entry for principal oryx/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab.
WARNING: no policy specified for oryx/127.29.159.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.159.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab.
Entry for principal oryx/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab.
I20260927 18:47:23.460019 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 6107
I20260927 18:47:23.465945 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 6241
I20260927 18:47:23.471130 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 6375
I20260927 18:47:23.476312 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 6013
I20260927 18:47:23.482273 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:39873
--webserver_interface=127.29.159.126
--webserver_port=43381
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:23.592906 6513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:23.593163 6513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:23.593235 6513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:23.597247 6513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:23.597307 6513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:23.597327 6513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:23.597388 6513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:23.597416 6513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:23.602674 6513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=43381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:23.603754 6513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:23.604769 6513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:23.612942 6519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:23.612954 6521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:23.612954 6518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:23.613279 6513 server_base.cc:1034] running on GCE node
I20260927 18:47:23.613845 6513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:23.614877 6513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:23.616060 6513 hybrid_clock.cc:648] HybridClock initialized: now 1790534843616040 us; error 38 us; skew 500 ppm
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534843, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:23.618706 6513 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:47:23.619603 6513 webserver.cc:460] Webserver started at http://127.29.159.126:43381/ using document root <none> and password file <none>
I20260927 18:47:23.620136 6513 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:23.620252 6513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:23.623888 6513 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 18:47:23.625874 6528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:23.627049 6513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:23.627184 6513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "5209703f3ee14be287083cfdf3516cf4"
format_stamp: "Formatted at 2026-09-27 18:47:22 on dist-test-slave-8tg2"
server_key: "16a801578e6d933a2fbec25d9d21ddae"
server_key_iv: "38178b3ad6d7d2f9f973bef02c9b8f81"
server_key_version: "encryptionkey@0"
I20260927 18:47:23.627791 6513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:23.648386 6513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:23.651134 6513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:23.651283 6513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:23.657943 6513 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:39873
I20260927 18:47:23.657955 6580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:39873 every 8 connection(s)
I20260927 18:47:23.658987 6513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:23.663090 6581 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:47:23.664734 6581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4: Bootstrap starting.
I20260927 18:47:23.668789 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 6513
I20260927 18:47:23.669289 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:38787
--local_ip_for_outbound_sockets=127.29.159.65
--tserver_master_addrs=127.29.159.126:39873
--webserver_port=37277
--webserver_interface=127.29.159.65
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.159.65
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:47:23.682184 6581 log.cc:824] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:23.688875 6581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:47:23.689261 6581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4: Bootstrap complete.
I20260927 18:47:23.693482 6581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } }
I20260927 18:47:23.693897 6581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5209703f3ee14be287083cfdf3516cf4, State: Initialized, Role: FOLLOWER
I20260927 18:47:23.694425 6581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } }
I20260927 18:47:23.694568 6581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:23.694654 6581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:23.694769 6581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:47:23.696771 6581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } }
I20260927 18:47:23.697104 6581 leader_election.cc:302] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5209703f3ee14be287083cfdf3516cf4; no voters:
I20260927 18:47:23.697450 6581 leader_election.cc:288] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:47:23.697543 6585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:47:23.697741 6585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [term 2 LEADER]: Becoming Leader. State: Replica: 5209703f3ee14be287083cfdf3516cf4, State: Running, Role: LEADER
I20260927 18:47:23.698032 6585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } }
I20260927 18:47:23.698715 6581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:23.699030 6587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5209703f3ee14be287083cfdf3516cf4. Latest consensus state: current_term: 2 leader_uuid: "5209703f3ee14be287083cfdf3516cf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } } }
I20260927 18:47:23.699157 6587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:23.699380 6586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5209703f3ee14be287083cfdf3516cf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209703f3ee14be287083cfdf3516cf4" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39873 } } }
I20260927 18:47:23.699497 6586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:23.699539 6589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:23.702808 6589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:23.703447 6589 catalog_manager.cc:1259] Loaded cluster ID: 54ddbb1dad064f2ab19ef17e990e397e
I20260927 18:47:23.703588 6589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:23.705946 6589 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:23.706583 6589 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5209703f3ee14be287083cfdf3516cf4: Loaded TSK: 0
I20260927 18:47:23.707859 6589 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:47:23.812810 6583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:23.813154 6583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:23.813217 6583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:23.819205 6583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:23.819300 6583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:23.819389 6583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:23.827046 6583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.65.keytab
--principal=oryx/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:38787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=37277
--tserver_master_addrs=127.29.159.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:23.828418 6583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:23.829717 6583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:23.840910 6610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:23.840893 6608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:23.840893 6607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:23.841549 6583 server_base.cc:1034] running on GCE node
I20260927 18:47:23.842031 6583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:23.842773 6583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:23.843971 6583 hybrid_clock.cc:648] HybridClock initialized: now 1790534843843935 us; error 52 us; skew 500 ppm
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534843, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:23.847299 6583 init.cc:377] Logged in from keytab as oryx/127.29.159.65@KRBTEST.COM (short username oryx)
I20260927 18:47:23.848332 6583 webserver.cc:460] Webserver started at http://127.29.159.65:37277/ using document root <none> and password file <none>
I20260927 18:47:23.848929 6583 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:23.849040 6583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:23.853152 6583 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:47:23.855515 6617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:23.857120 6583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:47:23.857270 6583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "fbb2389032d64e87ac8862d502210ec6"
format_stamp: "Formatted at 2026-09-27 18:47:22 on dist-test-slave-8tg2"
server_key: "78c98aadd56d84ac0c2134dfe5e28d1e"
server_key_iv: "707ee86d1295262567ecd1b170e81120"
server_key_version: "encryptionkey@0"
I20260927 18:47:23.857858 6583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:23.878182 6583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:23.881814 6583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:23.882036 6583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:23.882664 6583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:23.883610 6583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:23.883685 6583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:23.883754 6583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:23.883805 6583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:23.892763 6583 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:38787
I20260927 18:47:23.892838 6730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:38787 every 8 connection(s)
I20260927 18:47:23.893813 6583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:23.896697 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 6583
I20260927 18:47:23.897147 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:34303
--local_ip_for_outbound_sockets=127.29.159.66
--tserver_master_addrs=127.29.159.126:39873
--webserver_port=41827
--webserver_interface=127.29.159.66
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.159.66
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:23.909192 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:23.896199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:47587)
0927 18:47:23.896682 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:23.896735 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:23.898126 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:23.899539 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:23.899553 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:23.899907 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:23.900457 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:23.900472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.902533 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.902536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:23.903701 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:23.903711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.904894 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.904897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:23.905036 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:23.905694 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:23.905710 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:23.908362 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":157,"threads_started":1}
W20260927 18:47:23.909685 6731 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:39873 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.159.126:39873: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
W20260927 18:47:23.910132 6733 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:23.896013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:47587 (local address 127.29.159.126:39873)
0927 18:47:23.896453 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:23.896476 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:23.898475 (+ 1999us) server_negotiation.cc:407] Connection header received
0927 18:47:23.898706 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:23.898717 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:23.899016 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:23.899365 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:23.900656 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.902344 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.903832 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.904733 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.904941 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:23.908693 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:47587: BlockingRecv error: recv got EOF from 127.29.159.65:47587 (error 108)
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":143,"threads_started":1}
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:23.915864 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:23.910176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:58373)
0927 18:47:23.910533 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:23.910547 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:23.910647 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:23.910922 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:23.910924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:23.911052 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:23.911142 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:23.911147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.912203 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.912206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:23.912746 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:23.912753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.913309 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.913311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:23.913376 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:23.913893 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:23.913905 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:23.915767 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285}
W20260927 18:47:23.916052 6733 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:23.910245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:58373 (local address 127.29.159.126:39873)
0927 18:47:23.910431 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:23.910435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:23.910621 (+ 186us) server_negotiation.cc:407] Connection header received
0927 18:47:23.910745 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:23.910749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:23.910800 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:23.910886 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:23.911249 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.912098 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.912852 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.913205 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.913304 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:23.915930 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:58373: BlockingRecv error: recv got EOF from 127.29.159.65:58373 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":125,"threads_started":1}
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:23 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:23.921613 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:23.916392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:37669)
0927 18:47:23.916618 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:23.916635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:23.916722 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:23.916961 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:23.916963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:23.917081 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:23.917147 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:23.917151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.918140 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.918142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:23.918636 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:23.918642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.919180 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.919182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:23.919239 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:23.919743 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:23.919754 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:23.921514 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156}
W20260927 18:47:23.921818 6733 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:23.916498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:37669 (local address 127.29.159.126:39873)
0927 18:47:23.916620 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:23.916624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:23.916694 (+ 70us) server_negotiation.cc:407] Connection header received
0927 18:47:23.916813 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:23.916816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:23.916859 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:23.916939 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:23.917243 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.918022 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.918737 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:23.919074 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:23.919181 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:23.921685 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:37669: BlockingRecv error: recv got EOF from 127.29.159.65:37669 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 18:47:23.921839 6731 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:47:24.014771 6735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:24.015046 6735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:24.015098 6735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:24.019171 6735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:24.019245 6735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:24.019305 6735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:24.024907 6735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.66.keytab
--principal=oryx/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:34303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=41827
--tserver_master_addrs=127.29.159.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:24.026007 6735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:24.027019 6735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:24.037254 6744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:24.037220 6741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:24.037405 6735 server_base.cc:1034] running on GCE node
W20260927 18:47:24.037448 6742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:24.037837 6735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:24.038532 6735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:24.039732 6735 hybrid_clock.cc:648] HybridClock initialized: now 1790534844039715 us; error 30 us; skew 500 ppm
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534844, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:24.042580 6735 init.cc:377] Logged in from keytab as oryx/127.29.159.66@KRBTEST.COM (short username oryx)
I20260927 18:47:24.043555 6735 webserver.cc:460] Webserver started at http://127.29.159.66:41827/ using document root <none> and password file <none>
I20260927 18:47:24.044169 6735 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:24.044245 6735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:24.048115 6735 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:47:24.050133 6751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:24.051362 6735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:24.051540 6735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "bb150c4450a746cf9bfaef07a05daa7a"
format_stamp: "Formatted at 2026-09-27 18:47:23 on dist-test-slave-8tg2"
server_key: "7ab22a5fd449d40117a7fa3404a169be"
server_key_iv: "1b7c38e6553dc8279421c16fe7c4220d"
server_key_version: "encryptionkey@0"
I20260927 18:47:24.052078 6735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:24.077319 6735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:24.080976 6735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:24.081225 6735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:24.081876 6735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:24.082746 6735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:24.082799 6735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:24.082882 6735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:24.082921 6735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:24.092440 6735 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:34303
I20260927 18:47:24.092453 6864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:34303 every 8 connection(s)
I20260927 18:47:24.093462 6735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:24.094843 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 6735
I20260927 18:47:24.095320 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:36185
--local_ip_for_outbound_sockets=127.29.159.67
--tserver_master_addrs=127.29.159.126:39873
--webserver_port=43563
--webserver_interface=127.29.159.67
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.159.67
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:47:24.105758 6733 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:24.095415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:52739 (local address 127.29.159.126:39873)
0927 18:47:24.096782 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:24.096787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:24.096805 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:47:24.098107 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:24.098111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:24.098176 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:24.098280 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:24.099669 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.100292 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.101495 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.101797 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.101919 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.105595 (+ 3676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:52739: BlockingRecv error: recv got EOF from 127.29.159.66:52739 (error 108)
Metrics: {"server-negotiator.queue_time_us":1238}
I20260927 18:47:24.106405 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:24.095930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:52739)
0927 18:47:24.096485 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:24.096557 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:24.097863 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:24.098459 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:24.098467 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:24.098940 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:24.099526 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.099543 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.100418 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.100422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.101362 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.101372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.101961 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.101965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.102146 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.102917 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:24.102935 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:24.105469 (+ 2534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
W20260927 18:47:24.106912 6865 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:39873 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.159.126:39873: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:24.113686 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:24.107409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:35517)
0927 18:47:24.107598 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:24.107612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:24.107689 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:24.107932 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:24.107935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:24.108110 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:24.108222 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.108229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.109242 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.109246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.110282 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.110294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.110803 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.110807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.110910 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.111458 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:24.111476 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:24.113541 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122}
W20260927 18:47:24.113806 6733 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:24.107538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:35517 (local address 127.29.159.126:39873)
0927 18:47:24.107674 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:24.107677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:24.107690 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:47:24.107781 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:24.107783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:24.107822 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:24.107890 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:24.108410 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.109101 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.110423 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.110661 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.110775 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.113690 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:35517: BlockingRecv error: recv got EOF from 127.29.159.66:35517 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:24.119463 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:24.114461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:44649)
0927 18:47:24.114571 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:24.114585 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:24.114686 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:24.114924 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:24.114927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:24.115084 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:24.115161 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.115166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.115983 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.115989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.116507 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.116515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.116979 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.116982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.117062 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.117570 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:24.117581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:24.119310 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 18:47:24.119762 6733 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:24.114561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:44649 (local address 127.29.159.126:39873)
0927 18:47:24.114702 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:24.114708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:24.114727 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:47:24.114766 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:24.114769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:24.114822 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:24.114910 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:24.115282 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.115864 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.116637 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.116868 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.116980 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.119522 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:44649: BlockingRecv error: recv got EOF from 127.29.159.66:44649 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 18:47:24.119760 6865 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:47:24.211133 6868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:24.211433 6868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:24.211536 6868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:24.215553 6868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:24.215623 6868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:24.215713 6868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:24.220968 6868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/krb5kdc/oryx_127.29.159.67.keytab
--principal=oryx/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:36185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=43563
--tserver_master_addrs=127.29.159.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:24.221988 6868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:24.222992 6868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:24.232136 6874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:24.232208 6876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:24.232152 6873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:24.232542 6868 server_base.cc:1034] running on GCE node
I20260927 18:47:24.232916 6868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:24.233546 6868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:24.234728 6868 hybrid_clock.cc:648] HybridClock initialized: now 1790534844234700 us; error 46 us; skew 500 ppm
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534844, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:24.237370 6868 init.cc:377] Logged in from keytab as oryx/127.29.159.67@KRBTEST.COM (short username oryx)
I20260927 18:47:24.238241 6868 webserver.cc:460] Webserver started at http://127.29.159.67:43563/ using document root <none> and password file <none>
I20260927 18:47:24.238773 6868 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:24.238876 6868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:24.242560 6868 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:24.244585 6883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:24.245719 6868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:47:24.245848 6868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "c7a699639e9544d185114fafa12d43e8"
format_stamp: "Formatted at 2026-09-27 18:47:23 on dist-test-slave-8tg2"
server_key: "c6ff971bff125d900cd972f9cc482a5d"
server_key_iv: "2a79ae288ef32e9004a14559bc0365fe"
server_key_version: "encryptionkey@0"
I20260927 18:47:24.246374 6868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:24.266883 6868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:24.269855 6868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:24.270061 6868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:24.270646 6868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:24.271495 6868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:24.271567 6868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:24.271634 6868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:24.271672 6868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:24.280786 6868 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:36185
I20260927 18:47:24.280823 6996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:36185 every 8 connection(s)
I20260927 18:47:24.281746 6868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:24.282156 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 6868
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:24.292541 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:24.282933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.29.159.126:39873)
0927 18:47:24.283088 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:24.283094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:24.283288 (+ 194us) server_negotiation.cc:407] Connection header received
0927 18:47:24.283497 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:24.283501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:24.283555 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:24.283641 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:24.284530 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.285153 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.286391 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.286675 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.286754 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.288355 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:24.288362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:24.288378 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:24.288407 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:24.290320 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:24.290859 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:24.290863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:24.290867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:24.290974 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:24.291274 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:24.291278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:24.291280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:24.291649 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:24.291858 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:24.292122 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 18:47:24.292362 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:47:24.294023 6736 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:24.283846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:37191 (local address 127.29.159.126:39873)
0927 18:47:24.283996 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:24.284010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:24.284689 (+ 679us) server_negotiation.cc:407] Connection header received
0927 18:47:24.286319 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:24.286321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:24.286383 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:24.286468 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:24.287989 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.288623 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.290017 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.290366 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.290473 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.293929 (+ 3456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:37191: BlockingRecv error: recv got EOF from 127.29.159.67:37191 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:47:24.294791 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:24.284062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:37191)
0927 18:47:24.284548 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:24.284599 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:24.286101 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:24.286612 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:24.286619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:24.287128 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:24.287826 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.287850 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.288789 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.288795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.289889 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.289898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.290525 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.290532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.290746 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.291321 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:24.291335 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:24.293842 (+ 2507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
W20260927 18:47:24.295240 6997 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:39873 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.159.126:39873: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:24.299891 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:24.295630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:42239)
0927 18:47:24.295733 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:24.295748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:24.295814 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:24.295975 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:24.295977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:24.296092 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:24.296169 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.296173 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.296803 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.296805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.297328 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.297335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.297708 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.297711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.297778 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.298170 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:24.298180 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:24.299810 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 18:47:24.300073 6736 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:24.295697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:42239 (local address 127.29.159.126:39873)
0927 18:47:24.295803 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:24.295807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:24.295820 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:47:24.295873 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:24.295875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:24.295905 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:24.295961 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:24.296256 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.296717 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.297420 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.297618 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.297691 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.299930 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:42239: BlockingRecv error: recv got EOF from 127.29.159.67:42239 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:24.305047 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:24.300442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:40341)
0927 18:47:24.300561 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:24.300575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:24.300672 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:24.300894 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:24.300896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:24.301038 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:24.301151 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.301156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.301847 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.301852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.302368 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.302376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.302835 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.302838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.302918 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.303344 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:24.303356 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:24.304966 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 18:47:24.305190 6736 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:24.300535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:40341 (local address 127.29.159.126:39873)
0927 18:47:24.300681 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:24.300684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:24.300699 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:47:24.300741 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:24.300745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:24.300787 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:24.300873 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:24.301265 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.301737 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.302506 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.302719 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.302830 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.305083 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:40341: BlockingRecv error: recv got EOF from 127.29.159.67:40341 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 18:47:24.305349 6997 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:24 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:24.928069 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:24.922561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:37301)
0927 18:47:24.922858 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:24.922872 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:24.922973 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:24.923242 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:24.923244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:24.923568 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:24.923681 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.923686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.924425 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.924427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.925006 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:24.925013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.925430 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.925433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:24.925493 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.926027 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:24.926038 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:24.927945 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
W20260927 18:47:24.928308 7002 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:24.922666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:37301 (local address 127.29.159.126:39873)
0927 18:47:24.922929 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:24.922932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:24.922945 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:47:24.923067 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:24.923069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:24.923119 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:24.923201 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:24.923803 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.924331 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.925110 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:24.925333 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:24.925408 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:24.928097 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:37301: BlockingRecv error: recv got EOF from 127.29.159.65:37301 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
Sep 27 18:47:25 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:25 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:25.126035 7003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:25.120467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:60945)
0927 18:47:25.120754 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:25.120767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:25.120903 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:25.121168 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:25.121171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:25.121391 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:25.121489 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:25.121495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.122291 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.122296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:25.122879 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:25.122888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.123390 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.123394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:25.123532 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:25.124007 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:25.124019 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:25.125852 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
W20260927 18:47:25.126106 7002 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:25.120549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:60945 (local address 127.29.159.126:39873)
0927 18:47:25.120709 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:25.120714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:25.120842 (+ 128us) server_negotiation.cc:407] Connection header received
0927 18:47:25.121002 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:25.121005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:25.121052 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:25.121145 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:25.121614 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.122153 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.123018 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.123267 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.123375 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:25.125992 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:60945: BlockingRecv error: recv got EOF from 127.29.159.66:60945 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 18:47:25 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:25 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:25.311630 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:25.305948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:36735)
0927 18:47:25.306269 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:25.306286 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:25.306430 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:25.306705 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:25.306709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:25.306921 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:25.307014 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:25.307020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.307857 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.307862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:25.308486 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:25.308494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.309000 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.309004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:25.309097 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:25.309582 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:25.309595 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:25.311419 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":91,"threads_started":1}
W20260927 18:47:25.311841 7002 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:25.306058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:36735 (local address 127.29.159.126:39873)
0927 18:47:25.306246 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:25.306251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:25.306368 (+ 117us) server_negotiation.cc:407] Connection header received
0927 18:47:25.306533 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:25.306537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:25.306586 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:25.306680 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:25.307134 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.307744 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.308635 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.308875 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.308988 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:25.311610 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:36735: BlockingRecv error: recv got EOF from 127.29.159.67:36735 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 18:47:25 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:25 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:25.934432 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:25.929000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:56779)
0927 18:47:25.929294 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:25.929308 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:25.929398 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:25.929649 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:25.929652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:25.929830 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:25.929920 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:25.929925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.930767 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.930770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:25.931392 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:25.931406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.931882 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.931884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:25.931945 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:25.932549 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:25.932561 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:25.934322 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
W20260927 18:47:25.934655 7006 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:25.929050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:56779 (local address 127.29.159.126:39873)
0927 18:47:25.929333 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:25.929337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:25.929353 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:47:25.929486 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:25.929488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:25.929530 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:25.929622 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:25.930020 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.930650 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.931550 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:25.931783 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:25.931865 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:25.934480 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:56779: BlockingRecv error: recv got EOF from 127.29.159.65:56779 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":104,"threads_started":1}
Sep 27 18:47:26 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:26 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:26.132786 7007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:26.126833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:47613)
0927 18:47:26.127215 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:26.127238 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:26.127392 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:26.127753 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:26.127756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:26.127966 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:26.128057 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:26.128063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.128838 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.128844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:26.129415 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:26.129423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.129903 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.129907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:26.129994 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:26.130534 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:26.130545 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:26.132605 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":86,"threads_started":1}
W20260927 18:47:26.132862 7006 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:26.126967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:47613 (local address 127.29.159.126:39873)
0927 18:47:26.127148 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:26.127153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:26.127324 (+ 171us) server_negotiation.cc:407] Connection header received
0927 18:47:26.127546 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:26.127550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:26.127605 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:26.127733 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:26.128183 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.128719 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.129543 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.129782 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.129899 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:26.132751 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:47613: BlockingRecv error: recv got EOF from 127.29.159.66:47613 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 18:47:26 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:26 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:26.317718 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:26.312374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:54453)
0927 18:47:26.312667 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:26.312681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:26.312778 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:26.313013 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:26.313015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:26.313198 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:26.313295 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:26.313300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.314030 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.314034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:26.314641 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:26.314651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.315099 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.315103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:26.315168 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:26.315653 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:26.315665 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:26.317584 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":106,"threads_started":1}
W20260927 18:47:26.317885 7006 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:26.312510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:54453 (local address 127.29.159.126:39873)
0927 18:47:26.312644 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:26.312647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:26.312760 (+ 113us) server_negotiation.cc:407] Connection header received
0927 18:47:26.312858 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:26.312860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:26.312905 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:26.312972 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:26.313394 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.313922 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.314757 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.314959 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.315032 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:26.317721 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:54453: BlockingRecv error: recv got EOF from 127.29.159.67:54453 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 18:47:26 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:26 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:26.941108 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:26.935273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:40487)
0927 18:47:26.935577 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:26.935621 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:26.935730 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:26.935983 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:26.935985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:26.936177 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:26.936488 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:26.936502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.937259 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.937262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:26.937832 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:26.937840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.938286 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.938289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:26.938353 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:26.938865 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:26.938877 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:26.940920 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":103,"threads_started":1}
W20260927 18:47:26.941165 7010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:26.935361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:40487 (local address 127.29.159.126:39873)
0927 18:47:26.935646 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:26.935651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:26.935685 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:47:26.935823 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:26.935825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:26.935869 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:26.935946 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:26.936609 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.937139 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.937937 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:26.938167 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:26.938248 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:26.941042 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:40487: BlockingRecv error: recv got EOF from 127.29.159.65:40487 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":106,"threads_started":1}
Sep 27 18:47:27 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:27 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:27.138979 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:27.133557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:48373)
0927 18:47:27.133816 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:27.133830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:27.133932 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:27.134239 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:27.134241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:27.134419 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:27.134515 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:27.134519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.135274 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.135277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:27.135850 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:27.135858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.136327 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.136331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:27.136417 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:27.136986 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:27.136998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:27.138828 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
W20260927 18:47:27.139057 7010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:27.133619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:48373 (local address 127.29.159.126:39873)
0927 18:47:27.133751 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:27.133755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:27.133925 (+ 170us) server_negotiation.cc:407] Connection header received
0927 18:47:27.134024 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:27.134027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:27.134130 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:27.134197 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:27.134619 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.135179 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.135978 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.136209 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.136324 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:27.138966 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:48373: BlockingRecv error: recv got EOF from 127.29.159.66:48373 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 18:47:27 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:27 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:27.324730 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:27.318564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:42469)
0927 18:47:27.318903 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:27.318920 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:27.319070 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:27.319357 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:27.319360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:27.319684 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:27.319791 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:27.319797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.320647 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.320650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:27.321516 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:27.321527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.322161 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.322164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:27.322226 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:27.322720 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:27.322731 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:27.324577 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":86,"threads_started":1}
W20260927 18:47:27.324851 7010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:27.318714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:42469 (local address 127.29.159.126:39873)
0927 18:47:27.318896 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:27.318902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:27.319018 (+ 116us) server_negotiation.cc:407] Connection header received
0927 18:47:27.319189 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:27.319192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:27.319242 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:27.319335 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:27.319912 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.320496 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.321649 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.322008 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.322146 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:27.324714 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:42469: BlockingRecv error: recv got EOF from 127.29.159.67:42469 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 18:47:27 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:27 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:27.948338 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:27.942103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:55121)
0927 18:47:27.942382 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:27.942396 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:27.942484 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:27.942748 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:27.942750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:27.942933 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:27.943027 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:27.943032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.944114 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.944118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:27.944903 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:27.944916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.945423 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.945427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:27.945511 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:27.946101 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:27.946116 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:27.948169 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":110,"threads_started":1}
W20260927 18:47:27.948439 7014 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:27.942286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:55121 (local address 127.29.159.126:39873)
0927 18:47:27.942519 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:27.942523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:27.942535 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:47:27.942572 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:27.942574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:27.942620 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:27.942704 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:27.943185 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.943958 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.945022 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:27.945295 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:27.945418 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:27.948298 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:55121: BlockingRecv error: recv got EOF from 127.29.159.65:55121 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
Sep 27 18:47:28 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:28 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:28.145480 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:28.139852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:53529)
0927 18:47:28.140143 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:28.140160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:28.140322 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:28.140622 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:28.140625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:28.140820 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:28.140913 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:28.140918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.141688 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.141694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:28.142279 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:28.142305 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.142766 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.142769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:28.142853 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:28.143414 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:28.143449 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:28.145315 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
W20260927 18:47:28.145539 7014 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:28.140028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:53529 (local address 127.29.159.126:39873)
0927 18:47:28.140161 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:28.140166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:28.140258 (+ 92us) server_negotiation.cc:407] Connection header received
0927 18:47:28.140426 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:28.140430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:28.140483 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:28.140575 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:28.141036 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.141574 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.142423 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.142651 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.142763 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:28.145434 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:53529: BlockingRecv error: recv got EOF from 127.29.159.66:53529 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 18:47:28 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:28 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:28.346171 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:28.325443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:49375)
0927 18:47:28.325709 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:28.325723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:28.325822 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:28.326037 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:28.326039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:28.326214 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:28.326310 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:28.326315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.327021 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.327023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:28.327639 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:28.327648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.328053 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.328055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:28.328134 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:28.344167 (+ 16033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:28.344181 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:28.346033 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
W20260927 18:47:28.346377 7014 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:28.325499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:49375 (local address 127.29.159.126:39873)
0927 18:47:28.325635 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:28.325639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:28.325791 (+ 152us) server_negotiation.cc:407] Connection header received
0927 18:47:28.325887 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:28.325889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:28.325933 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:28.325999 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:28.326408 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.326918 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.327744 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.327952 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.328029 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:28.346203 (+ 18174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:49375: BlockingRecv error: recv got EOF from 127.29.159.67:49375 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 18:47:28 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:28 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:28.955642 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:28.949358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:37165)
0927 18:47:28.949672 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:28.949686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:28.949784 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:28.950034 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:28.950036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:28.950237 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:28.950333 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:28.950339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.951391 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.951394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:28.952034 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:28.952045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.952542 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.952545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:28.952631 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:28.953252 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:28.953264 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:28.955467 (+ 2203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":110,"threads_started":1}
W20260927 18:47:28.955713 7018 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:28.949420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:37165 (local address 127.29.159.126:39873)
0927 18:47:28.949683 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:28.949687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:28.949758 (+ 71us) server_negotiation.cc:407] Connection header received
0927 18:47:28.949864 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:28.949867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:28.949911 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:28.950025 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:28.950440 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.951260 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.952186 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:28.952401 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:28.952486 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:28.955589 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:37165: BlockingRecv error: recv got EOF from 127.29.159.65:37165 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":117,"threads_started":1}
Sep 27 18:47:29 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:29 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:29.152074 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:29.146346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:49075)
0927 18:47:29.146683 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:29.146699 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:29.146814 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:29.147077 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:29.147080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:29.147313 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:29.147404 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:29.147409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.148190 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.148196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:29.148797 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:29.148805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.149256 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.149267 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:29.149352 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:29.149873 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:29.149883 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:29.151905 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
W20260927 18:47:29.152118 7018 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:29.146533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:49075 (local address 127.29.159.126:39873)
0927 18:47:29.146685 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:29.146691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:29.146769 (+ 78us) server_negotiation.cc:407] Connection header received
0927 18:47:29.146914 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:29.146917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:29.146965 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:29.147056 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:29.147548 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.148080 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.148915 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.149137 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.149256 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:29.152020 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:49075: BlockingRecv error: recv got EOF from 127.29.159.66:49075 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 18:47:29 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:29 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:29.353653 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:29.347110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:57981)
0927 18:47:29.347477 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:29.347494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:29.347611 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:29.347972 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:29.347977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:29.348258 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:29.348378 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:29.348384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.349379 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.349382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:29.350302 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:29.350314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.350755 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.350758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:29.350822 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:29.351463 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:29.351483 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:29.353503 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":125,"threads_started":1}
W20260927 18:47:29.353866 7018 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:29.347270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:57981 (local address 127.29.159.126:39873)
0927 18:47:29.347460 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:29.347463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:29.347555 (+ 92us) server_negotiation.cc:407] Connection header received
0927 18:47:29.347690 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:29.347692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:29.347738 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:29.347841 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:29.348533 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.349257 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.350417 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.350656 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.350734 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:29.353672 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:57981: BlockingRecv error: recv got EOF from 127.29.159.67:57981 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 18:47:29 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:29 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:29.963871 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:29.956601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:34803)
0927 18:47:29.956949 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:29.956970 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:29.957107 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:29.957402 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:29.957406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:29.957661 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:29.957797 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:29.957807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.958709 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.958714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:29.959905 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:29.959920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.960625 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.960633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:29.960760 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:29.961503 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:29.961522 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:29.963667 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
W20260927 18:47:29.963932 7022 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:29.956713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:34803 (local address 127.29.159.126:39873)
0927 18:47:29.956980 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:29.956984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:29.957040 (+ 56us) server_negotiation.cc:407] Connection header received
0927 18:47:29.957214 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:29.957217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:29.957264 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:29.957346 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:29.957971 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.958579 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.960065 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:29.960383 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:29.960500 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:29.963802 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:34803: BlockingRecv error: recv got EOF from 127.29.159.65:34803 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":56,"threads_started":1}
Sep 27 18:47:30 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:30 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:30.158670 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:30.153117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:39001)
0927 18:47:30.153457 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:30.153471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:30.153601 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:30.153886 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:30.153889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:30.154111 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:30.154201 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:30.154206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.154981 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.154988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:30.155752 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:30.155762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.156215 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.156219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:30.156297 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:30.156795 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:30.156806 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:30.158552 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":106,"threads_started":1}
W20260927 18:47:30.158806 7022 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:30.153234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:39001 (local address 127.29.159.126:39873)
0927 18:47:30.153425 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:30.153430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:30.153551 (+ 121us) server_negotiation.cc:407] Connection header received
0927 18:47:30.153718 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:30.153721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:30.153779 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:30.153865 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:30.154319 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.154858 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.155869 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.156102 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.156212 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:30.158684 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:39001: BlockingRecv error: recv got EOF from 127.29.159.66:39001 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 18:47:30 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:30 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:30.360532 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:30.354682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:42979)
0927 18:47:30.354981 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:30.354995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:30.355098 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:30.355328 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:30.355330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:30.355599 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:30.355754 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:30.355761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.356456 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.356459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:30.357218 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:30.357243 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.357802 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.357809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:30.357899 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:30.358522 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:30.358534 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:30.360419 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":100,"threads_started":1}
W20260927 18:47:30.360656 7022 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:30.354803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:42979 (local address 127.29.159.126:39873)
0927 18:47:30.354985 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:30.354988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:30.355050 (+ 62us) server_negotiation.cc:407] Connection header received
0927 18:47:30.355172 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:30.355174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:30.355221 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:30.355292 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:30.355855 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.356371 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.357354 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.357644 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.357789 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:30.360541 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:42979: BlockingRecv error: recv got EOF from 127.29.159.67:42979 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 18:47:30 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:30 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:30.970692 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:30.964950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:58963)
0927 18:47:30.965361 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:30.965377 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:30.965487 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:30.965755 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:30.965757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:30.965936 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:30.966044 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:30.966050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.966815 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.966818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:30.967384 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:30.967392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.967915 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.967919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:30.968018 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:30.968586 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:30.968597 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:30.970573 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":97,"threads_started":1}
W20260927 18:47:30.970908 7026 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:30.965022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:58963 (local address 127.29.159.126:39873)
0927 18:47:30.965272 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:30.965277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:30.965458 (+ 181us) server_negotiation.cc:407] Connection header received
0927 18:47:30.965579 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:30.965581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:30.965628 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:30.965709 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:30.966156 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.966706 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.967552 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:30.967800 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:30.967911 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:30.970754 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:58963: BlockingRecv error: recv got EOF from 127.29.159.65:58963 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
Sep 27 18:47:31 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:31 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:31.165259 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:31.159767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:57687)
0927 18:47:31.160171 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:31.160186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:31.160326 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:31.160619 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:31.160621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:31.160801 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:31.160893 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:31.160898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.161635 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.161638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:31.162330 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:31.162344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.162734 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.162736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:31.162798 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:31.163272 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:31.163284 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:31.165149 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":86,"threads_started":1}
W20260927 18:47:31.165405 7026 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:31.159934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:57687 (local address 127.29.159.126:39873)
0927 18:47:31.160090 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:31.160097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:31.160299 (+ 202us) server_negotiation.cc:407] Connection header received
0927 18:47:31.160426 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:31.160429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:31.160494 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:31.160570 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:31.161022 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.161545 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.162431 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.162641 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.162719 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:31.165287 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:57687: BlockingRecv error: recv got EOF from 127.29.159.66:57687 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 18:47:31 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:31 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:31.367203 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:31.361394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:56645)
0927 18:47:31.361760 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:31.361773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:31.361903 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:31.362197 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:31.362200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:31.362388 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:31.362484 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:31.362489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.363330 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.363333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:31.363981 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:31.363989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.364492 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.364494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:31.364557 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:31.365158 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:31.365171 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:31.367086 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":114,"threads_started":1}
W20260927 18:47:31.367553 7026 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:31.361504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:56645 (local address 127.29.159.126:39873)
0927 18:47:31.361653 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:31.361660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:31.361853 (+ 193us) server_negotiation.cc:407] Connection header received
0927 18:47:31.361984 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:31.361986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:31.362035 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:31.362159 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:31.362584 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.363194 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.364116 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.364387 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.364476 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:31.367258 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:56645: BlockingRecv error: recv got EOF from 127.29.159.67:56645 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 18:47:31 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:31 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:31.977187 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:31.971553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:49719)
0927 18:47:31.971913 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:31.971931 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:31.972036 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:31.972316 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:31.972318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:31.972496 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:31.972590 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:31.972595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.973401 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.973408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:31.974037 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:31.974045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.974538 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.974541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:31.974599 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:31.975163 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:31.975178 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:31.977060 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":80,"threads_started":1}
W20260927 18:47:31.977403 7030 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:31.971693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:49719 (local address 127.29.159.126:39873)
0927 18:47:31.971959 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:31.971963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:31.971985 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:47:31.972148 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:31.972152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:31.972210 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:31.972302 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:31.972694 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.973270 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.974203 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:31.974433 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:31.974525 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:31.977249 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:49719: BlockingRecv error: recv got EOF from 127.29.159.65:49719 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":66,"threads_started":1}
Sep 27 18:47:32 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:32 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:32.172159 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:32.166050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:33161)
0927 18:47:32.166346 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:32.166361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:32.166462 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:32.166688 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:32.166690 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:32.166866 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:32.166954 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:32.166960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.167933 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.167935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:32.168486 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:32.168494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.168960 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.168962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:32.169026 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:32.169605 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:32.169616 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:32.171991 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":98,"threads_started":1}
W20260927 18:47:32.172242 7030 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:32.166100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:33161 (local address 127.29.159.126:39873)
0927 18:47:32.166254 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:32.166259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:32.166466 (+ 207us) server_negotiation.cc:407] Connection header received
0927 18:47:32.166534 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:32.166536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:32.166583 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:32.166663 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:32.167055 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.167834 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.168616 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.168850 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.168944 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:32.172127 (+ 3183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:33161: BlockingRecv error: recv got EOF from 127.29.159.66:33161 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 18:47:32 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:32 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:32.373915 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:32.368192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:39887)
0927 18:47:32.368512 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:32.368527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:32.368685 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:32.369009 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:32.369012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:32.369273 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:32.369418 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:32.369426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.370301 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.370304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:32.370879 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:32.370887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.371314 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.371317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:32.371383 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:32.371950 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:32.371962 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:32.373799 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":107,"threads_started":1}
W20260927 18:47:32.374111 7030 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:32.368361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:39887 (local address 127.29.159.126:39873)
0927 18:47:32.368529 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:32.368534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:32.368636 (+ 102us) server_negotiation.cc:407] Connection header received
0927 18:47:32.368797 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:32.368801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:32.368861 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:32.368966 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:32.369550 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.370170 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.370985 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.371201 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.371281 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:32.373971 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:39887: BlockingRecv error: recv got EOF from 127.29.159.67:39887 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 18:47:32 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:32 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:32.984323 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:32.978140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:57073)
0927 18:47:32.978428 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:32.978443 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:32.978540 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:32.978886 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:32.978889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:32.979072 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:32.979213 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:32.979222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.980310 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.980317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:32.981063 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:32.981073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.981534 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.981536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:32.981597 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:32.982123 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:32.982135 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:32.984185 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
W20260927 18:47:32.984462 7034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:32.978154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:57073 (local address 127.29.159.126:39873)
0927 18:47:32.978471 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:32.978475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:32.978492 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:47:32.978684 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:32.978687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:32.978735 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:32.978822 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:32.979356 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.980169 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.981199 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:32.981427 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:32.981520 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:32.984326 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:57073: BlockingRecv error: recv got EOF from 127.29.159.65:57073 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":120,"threads_started":1}
Sep 27 18:47:33 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:33 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:33.179330 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:33.173255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:36887)
0927 18:47:33.173551 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:33.173578 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:33.173719 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:33.173991 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:33.173994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:33.174202 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:33.174293 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:33.174299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.175078 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.175082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:33.175709 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:33.175717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.176235 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.176239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:33.176323 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:33.176885 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:33.176897 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:33.179211 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":78,"threads_started":1}
W20260927 18:47:33.179544 7034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:33.173331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:36887 (local address 127.29.159.126:39873)
0927 18:47:33.173520 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:33.173525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:33.173660 (+ 135us) server_negotiation.cc:407] Connection header received
0927 18:47:33.173821 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:33.173825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:33.173878 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:33.173970 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:33.174412 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.174960 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.175830 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.176111 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.176222 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:33.179355 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:36887: BlockingRecv error: recv got EOF from 127.29.159.66:36887 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 18:47:33 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:33 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:33.388257 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:33.374793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:48353)
0927 18:47:33.375082 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:33.375099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:33.375208 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:33.375512 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:33.375514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:33.375724 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:33.375822 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:33.375827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.376763 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.376768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:33.377339 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:33.377349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.377911 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.377915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:33.378002 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:33.385824 (+ 7822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:33.385838 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:33.388123 (+ 2285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
W20260927 18:47:33.388489 7034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:33.374938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:48353 (local address 127.29.159.126:39873)
0927 18:47:33.375111 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:33.375115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:33.375173 (+ 58us) server_negotiation.cc:407] Connection header received
0927 18:47:33.375290 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:33.375292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:33.375338 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:33.375410 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:33.375923 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.376640 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.377470 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.377789 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.377903 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:33.388351 (+ 10448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:48353: BlockingRecv error: recv got EOF from 127.29.159.67:48353 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 18:47:33 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:33 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:33.990723 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:33.985141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:49685)
0927 18:47:33.985411 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:33.985425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:33.985512 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:33.985766 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:33.985768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:33.985945 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:33.986040 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:33.986045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.986844 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.986846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:33.987479 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:33.987492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.987974 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.987976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:33.988037 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:33.988556 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:33.988567 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:33.990593 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
W20260927 18:47:33.990929 7038 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:33.985308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:49685 (local address 127.29.159.126:39873)
0927 18:47:33.985549 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:33.985553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:33.985565 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:47:33.985609 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:33.985610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:33.985651 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:33.985732 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:33.986188 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.986745 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.987622 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:33.987869 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:33.987960 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:33.990765 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:49685: BlockingRecv error: recv got EOF from 127.29.159.65:49685 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
Sep 27 18:47:34 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:34 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:34.185576 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:34.180131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:33971)
0927 18:47:34.180432 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:34.180451 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:34.180575 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:34.180885 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:34.180888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:34.181094 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:34.181184 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:34.181191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.181976 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.181981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:34.182541 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:34.182549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.183013 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.183016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:34.183100 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:34.183637 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:34.183648 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:34.185407 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":88,"threads_started":1}
W20260927 18:47:34.185638 7038 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:34.180235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:33971 (local address 127.29.159.126:39873)
0927 18:47:34.180449 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:34.180454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:34.180523 (+ 69us) server_negotiation.cc:407] Connection header received
0927 18:47:34.180700 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:34.180702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:34.180758 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:34.180857 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:34.181324 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.181862 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.182663 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.182896 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.183002 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:34.185533 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:33971: BlockingRecv error: recv got EOF from 127.29.159.66:33971 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 18:47:34 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:34 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:34.395174 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:34.389220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:34005)
0927 18:47:34.389573 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:34.389588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:34.389686 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:34.389987 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:34.389989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:34.390169 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:34.390285 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:34.390290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.391175 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.391181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:34.391912 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:34.391921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.392402 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.392406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:34.392491 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:34.392998 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:34.393009 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:34.395048 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
W20260927 18:47:34.395304 7038 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:34.389444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:34005 (local address 127.29.159.126:39873)
0927 18:47:34.389596 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:34.389600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:34.389640 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:47:34.389806 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:34.389809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:34.389860 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:34.389947 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:34.390420 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.391063 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.392040 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.392278 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.392392 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:34.395186 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:34005: BlockingRecv error: recv got EOF from 127.29.159.67:34005 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 18:47:34 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:34 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:34.997787 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:34.991597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:44431)
0927 18:47:34.992108 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:34.992130 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:34.992226 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:34.992564 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:34.992569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:34.992772 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:34.992870 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:34.992875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.993733 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.993736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:34.994294 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:34.994302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.994799 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.994802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:34.994861 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:34.995459 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:34.995470 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:34.997596 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":84,"threads_started":1}
W20260927 18:47:34.997871 7042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:34.991656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:44431 (local address 127.29.159.126:39873)
0927 18:47:34.991930 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:34.991934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:34.992218 (+ 284us) server_negotiation.cc:407] Connection header received
0927 18:47:34.992340 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:34.992342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:34.992388 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:34.992506 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:34.993007 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.993605 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.994464 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:34.994693 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:34.994781 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:34.997734 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:44431: BlockingRecv error: recv got EOF from 127.29.159.65:44431 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
Sep 27 18:47:35 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:35 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:35.191923 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:35.186398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:47551)
0927 18:47:35.186670 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:35.186683 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:35.186785 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:35.187057 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:35.187061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:35.187261 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:35.187377 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:35.187383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:35.188160 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:35.188163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:35.188732 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:35.188739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:35.189157 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:35.189160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:35.189220 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:35.189708 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:35.189720 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:35.191789 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
W20260927 18:47:35.192075 7042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:35.186469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:47551 (local address 127.29.159.126:39873)
0927 18:47:35.186618 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:35.186622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:35.186756 (+ 134us) server_negotiation.cc:407] Connection header received
0927 18:47:35.186867 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:35.186869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:35.186913 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:35.186985 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:35.187531 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:35.188038 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:35.188832 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:35.189037 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:35.189110 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:35.191908 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:47551: BlockingRecv error: recv got EOF from 127.29.159.66:47551 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 18:47:35 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:35 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:35.401649 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:35.395976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:49125)
0927 18:47:35.396247 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:35.396261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:35.396360 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:35.396603 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:35.396605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:35.396806 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:35.396903 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:35.396909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:35.397626 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:35.397629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:35.398218 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:35.398226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:35.398653 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:35.398656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:35.398735 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:35.399335 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:35.399347 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:35.401493 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
W20260927 18:47:35.401796 7042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:35.396033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:49125 (local address 127.29.159.126:39873)
0927 18:47:35.396185 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:35.396189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:35.396336 (+ 147us) server_negotiation.cc:407] Connection header received
0927 18:47:35.396437 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:35.396439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:35.396488 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:35.396564 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:35.397005 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:35.397527 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:35.398319 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:35.398527 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:35.398619 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:35.401631 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:49125: BlockingRecv error: recv got EOF from 127.29.159.67:49125 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 18:47:35 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:35 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:36.004737 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:35.998725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:54901)
0927 18:47:35.999006 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:35.999024 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:35.999201 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:35.999588 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:35.999590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:35.999840 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:35.999953 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:35.999959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.000841 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.000844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:36.001493 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:36.001503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.001975 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.001978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:36.002039 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:36.002604 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:36.002616 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:36.004615 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
W20260927 18:47:36.004894 7046 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:35.998863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:54901 (local address 127.29.159.126:39873)
0927 18:47:35.999180 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:35.999187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:35.999207 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:47:35.999388 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:35.999391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:35.999463 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:35.999552 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:36.000069 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.000722 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.001624 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.001863 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.001955 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:36.004754 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:54901: BlockingRecv error: recv got EOF from 127.29.159.65:54901 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
Sep 27 18:47:36 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:36 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:36.198284 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:36.192746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:43189)
0927 18:47:36.193030 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:36.193044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:36.193160 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:36.193433 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:36.193436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:36.193646 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:36.193750 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:36.193757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.194507 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.194510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:36.195135 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:36.195144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.195636 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.195640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:36.195719 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:36.196257 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:36.196269 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:36.198163 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":112,"threads_started":1}
W20260927 18:47:36.198474 7046 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:36.192840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:43189 (local address 127.29.159.126:39873)
0927 18:47:36.192987 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:36.192991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:36.193101 (+ 110us) server_negotiation.cc:407] Connection header received
0927 18:47:36.193245 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:36.193247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:36.193289 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:36.193348 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:36.193865 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.194393 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.195242 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.195472 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.195548 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:36.198312 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:43189: BlockingRecv error: recv got EOF from 127.29.159.66:43189 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 18:47:36 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:36 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:36.408391 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:36.402452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:33937)
0927 18:47:36.402758 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:36.402774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:36.402923 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:36.403196 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:36.403199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:36.403446 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:36.403560 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:36.403565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.404329 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.404334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:36.404904 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:36.404912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.405371 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.405375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:36.405462 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:36.405994 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:36.406010 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:36.408276 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":107,"threads_started":1}
W20260927 18:47:36.408651 7046 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:36.402533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:33937 (local address 127.29.159.126:39873)
0927 18:47:36.402704 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:36.402709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:36.402859 (+ 150us) server_negotiation.cc:407] Connection header received
0927 18:47:36.403029 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:36.403032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:36.403084 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:36.403174 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:36.403678 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.404208 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.405023 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:36.405253 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:36.405365 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:36.408399 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:33937: BlockingRecv error: recv got EOF from 127.29.159.67:33937 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 18:47:37 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:37 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:37.011338 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:37.005609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:37657)
0927 18:47:37.005920 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:37.005934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:37.006050 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:37.006319 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:37.006321 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:37.006494 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:37.006587 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:37.006592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.007331 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.007333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:37.007986 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:37.007995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.008431 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.008433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:37.008490 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:37.009122 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:37.009135 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:37.011166 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":87,"threads_started":1}
W20260927 18:47:37.011413 7050 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:37.005670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:37657 (local address 127.29.159.126:39873)
0927 18:47:37.005924 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:37.005928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:37.006009 (+ 81us) server_negotiation.cc:407] Connection header received
0927 18:47:37.006131 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:37.006133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:37.006208 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:37.006288 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:37.006684 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.007232 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.008083 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.008330 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.008417 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:37.011284 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:37657: BlockingRecv error: recv got EOF from 127.29.159.65:37657 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":59,"threads_started":1}
Sep 27 18:47:37 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:37 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:37.204471 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:37.199167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:54623)
0927 18:47:37.199460 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:37.199474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:37.199564 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:37.199786 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:37.199788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:37.200001 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:37.200177 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:37.200189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.200997 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.201000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:37.201547 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:37.201555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.201992 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.201995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:37.202054 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:37.202545 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:37.202557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:37.204340 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
W20260927 18:47:37.204620 7050 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:37.199346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:54623 (local address 127.29.159.126:39873)
0927 18:47:37.199521 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:37.199524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:37.199538 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:47:37.199633 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:37.199635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:37.199676 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:37.199749 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:37.200333 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.200885 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.201660 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.201893 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.201974 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:37.204469 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:54623: BlockingRecv error: recv got EOF from 127.29.159.66:54623 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 18:47:37 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:37 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:37.414788 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:37.409197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:33403)
0927 18:47:37.409452 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:37.409467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:37.409572 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:37.409814 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:37.409816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:37.410010 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:37.410161 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:37.410167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.410894 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.410897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:37.411624 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:37.411635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.412138 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.412141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:37.412204 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:37.412719 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:37.412730 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:37.414646 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
W20260927 18:47:37.414940 7050 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:37.409262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:33403 (local address 127.29.159.126:39873)
0927 18:47:37.409413 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:37.409420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:37.409540 (+ 120us) server_negotiation.cc:407] Connection header received
0927 18:47:37.409654 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:37.409656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:37.409704 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:37.409770 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:37.410268 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.410791 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.411758 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:37.411989 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:37.412130 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:37.414801 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:33403: BlockingRecv error: recv got EOF from 127.29.159.67:33403 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 18:47:38 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:38 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:38.017915 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:38.012341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:47339)
0927 18:47:38.012630 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:38.012645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:38.012745 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:38.013054 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:38.013056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:38.013250 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:38.013344 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:38.013349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.014106 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.014108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:38.014673 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:38.014680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.015245 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.015247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:38.015305 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:38.015871 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:38.015883 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:38.017766 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
W20260927 18:47:38.018142 7054 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:38.012403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:47339 (local address 127.29.159.126:39873)
0927 18:47:38.012701 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:38.012704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:38.012718 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:47:38.012880 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:38.012882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:38.012931 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:38.013018 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:38.013472 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.014010 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.014851 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.015077 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.015218 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:38.017905 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:47339: BlockingRecv error: recv got EOF from 127.29.159.65:47339 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":59,"threads_started":1}
Sep 27 18:47:38 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:38 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:38.211078 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:38.205286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:56039)
0927 18:47:38.205624 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:38.205638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:38.205785 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:38.206146 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:38.206149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:38.206433 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:38.206563 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:38.206574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.207526 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.207529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:38.208100 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:38.208108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.208531 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.208534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:38.208595 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:38.209082 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:38.209093 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:38.210960 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":86,"threads_started":1}
W20260927 18:47:38.211230 7054 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:38.205417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:56039 (local address 127.29.159.126:39873)
0927 18:47:38.205599 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:38.205604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:38.205747 (+ 143us) server_negotiation.cc:407] Connection header received
0927 18:47:38.205939 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:38.205943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:38.205999 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:38.206101 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:38.206720 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.207412 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.208215 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.208430 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.208510 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:38.211105 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:56039: BlockingRecv error: recv got EOF from 127.29.159.66:56039 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 18:47:38 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:38 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.67@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:38.429507 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:38.415742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.67:45457)
0927 18:47:38.416061 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:38.416077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:38.416226 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:38.416615 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:38.416618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:38.416839 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:38.416935 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:38.416941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.417774 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.417777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:38.418389 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:38.418398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.418847 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.418850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:38.418920 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:38.427073 (+ 8153us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:38.427091 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:38.429380 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":84,"threads_started":1}
W20260927 18:47:38.429752 7054 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:38.415870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:45457 (local address 127.29.159.126:39873)
0927 18:47:38.416039 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:38.416044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:38.416165 (+ 121us) server_negotiation.cc:407] Connection header received
0927 18:47:38.416356 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:38.416359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:38.416419 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:38.416508 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:38.417055 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.417659 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.418496 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:38.418725 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:38.418801 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:38.429525 (+ 10724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.67:45457: BlockingRecv error: recv got EOF from 127.29.159.67:45457 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 18:47:39 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:39 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.65@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:39.025077 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:39.019096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.65:40569)
0927 18:47:39.019455 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:39.019472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:39.019599 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:39.019962 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:39.019966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:39.020160 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:39.020280 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:39.020287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.021123 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.021125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:39.021706 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:39.021715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.022193 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.022195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:39.022274 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:39.022829 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:39.022840 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:39.024960 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":93,"threads_started":1}
W20260927 18:47:39.025350 7058 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:39.019155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:40569 (local address 127.29.159.126:39873)
0927 18:47:39.019462 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:39.019466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:39.019711 (+ 245us) server_negotiation.cc:407] Connection header received
0927 18:47:39.019752 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:39.019754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:39.019809 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:39.019914 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:39.020409 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.021015 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.021849 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.022083 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.022177 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:39.025165 (+ 2988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.65:40569: BlockingRecv error: recv got EOF from 127.29.159.65:40569 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":68,"threads_started":1}
Sep 27 18:47:39 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:47:39 dist-test-slave-8tg2 krb5kdc[6000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.159.66@KRBTEST.COM for oryx/127.29.159.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:47:39.217842 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:39.212129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39873 (local address 127.29.159.66:37793)
0927 18:47:39.212487 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:39.212501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:39.212674 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:39.212972 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:39.212975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:39.213193 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:39.213283 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:39.213289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.214074 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.214078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:39.214673 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:39.214681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.215157 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.215160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:39.215248 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:39.215754 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:39.215765 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:39.217673 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.159.126:39873: Server oryx/127.29.159.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
W20260927 18:47:39.217896 7058 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:39.212260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:37793 (local address 127.29.159.126:39873)
0927 18:47:39.212439 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:39.212444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:39.212611 (+ 167us) server_negotiation.cc:407] Connection header received
0927 18:47:39.212792 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:39.212795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:39.212846 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:39.212947 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:39.213407 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.213962 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.214797 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.215030 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.215155 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:39.217791 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.159.66:37793: BlockingRecv error: recv got EOF from 127.29.159.66:37793 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 18:47:39.291963 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 6583
I20260927 18:47:39.297643 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 6735
I20260927 18:47:39.304848 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 6868
I20260927 18:47:39.310132 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 6513
2026-09-27T18:47:39Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17993 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T18:47:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:39Z Disabled control of system clock
I20260927 18:47:39.340329 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:39529
--webserver_interface=127.29.159.126
--webserver_port=0
--builtin_ntp_servers=127.29.159.84:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:39529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:47:39.449175 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:39.449434 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:39.449468 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:39.453534 7063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:39.453593 7063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:39.453609 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:39.453626 7063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:39.453641 7063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:39.458449 7063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:39529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:39529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:39.459553 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:39.460945 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:39.469602 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:39.469615 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:39.469615 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:39.469765 7063 server_base.cc:1034] running on GCE node
I20260927 18:47:39.470439 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:39.471601 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:39.472777 7063 hybrid_clock.cc:648] HybridClock initialized: now 1790534859472736 us; error 53 us; skew 500 ppm
I20260927 18:47:39.474738 7063 webserver.cc:460] Webserver started at http://127.29.159.126:41859/ using document root <none> and password file <none>
I20260927 18:47:39.475344 7063 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:39.475406 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:39.475677 7063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:39.477813 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "52ea22788aab40d48b277db8af6bf8a5"
format_stamp: "Formatted at 2026-09-27 18:47:39 on dist-test-slave-8tg2"
server_key: "6fceb264c57d7574a566cdf37eb12aa9"
server_key_iv: "6ec364e48c36b89b04c1da37248112ec"
server_key_version: "encryptionkey@0"
I20260927 18:47:39.478434 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "52ea22788aab40d48b277db8af6bf8a5"
format_stamp: "Formatted at 2026-09-27 18:47:39 on dist-test-slave-8tg2"
server_key: "6fceb264c57d7574a566cdf37eb12aa9"
server_key_iv: "6ec364e48c36b89b04c1da37248112ec"
server_key_version: "encryptionkey@0"
I20260927 18:47:39.482267 7063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:47:39.484900 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:39.486567 7063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:47:39.486697 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "52ea22788aab40d48b277db8af6bf8a5"
format_stamp: "Formatted at 2026-09-27 18:47:39 on dist-test-slave-8tg2"
server_key: "6fceb264c57d7574a566cdf37eb12aa9"
server_key_iv: "6ec364e48c36b89b04c1da37248112ec"
server_key_version: "encryptionkey@0"
I20260927 18:47:39.486817 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:39.513201 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:39.513893 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:39.514050 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:39.521152 7063 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:39529
I20260927 18:47:39.521162 7129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:39529 every 8 connection(s)
I20260927 18:47:39.522140 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:39.525503 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:39.526086 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 7063
I20260927 18:47:39.526199 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:39.526562 30333 external_mini_cluster.cc:1428] Setting key 45e4984eef575f5e8f4ce7d9549b0083
I20260927 18:47:39.531957 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5: Bootstrap starting.
I20260927 18:47:39.534365 7130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:39.535125 7130 log.cc:824] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:39.536271 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:39.528111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.29.159.126:39529)
0927 18:47:39.528816 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:39.528831 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:39.528869 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:47:39.529513 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:39.529528 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:39.529947 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:39.530359 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:39.530816 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.531858 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.532784 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.533297 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.533496 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:39.534460 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:39.534468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:39.534489 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:39.534928 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:39.534987 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:39.535007 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:39.535126 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:39.535246 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 18:47:39.535557 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":139,"threads_started":1}
I20260927 18:47:39.537420 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5: No bootstrap required, opened a new log
I20260927 18:47:39.541173 7130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ea22788aab40d48b277db8af6bf8a5" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39529 } }
I20260927 18:47:39.541436 7130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:39.541522 7130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ea22788aab40d48b277db8af6bf8a5, State: Initialized, Role: FOLLOWER
I20260927 18:47:39.542011 7130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ea22788aab40d48b277db8af6bf8a5" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39529 } }
I20260927 18:47:39.542167 7130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:39.542256 7130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:39.542357 7130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:39.543380 7130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ea22788aab40d48b277db8af6bf8a5" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39529 } }
I20260927 18:47:39.543800 7130 leader_election.cc:302] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52ea22788aab40d48b277db8af6bf8a5; no voters:
I20260927 18:47:39.544142 7130 leader_election.cc:288] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:39.544274 7135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:39.544526 7135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [term 1 LEADER]: Becoming Leader. State: Replica: 52ea22788aab40d48b277db8af6bf8a5, State: Running, Role: LEADER
I20260927 18:47:39.544951 7135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ea22788aab40d48b277db8af6bf8a5" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39529 } }
I20260927 18:47:39.545408 7130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:39.546618 7136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52ea22788aab40d48b277db8af6bf8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ea22788aab40d48b277db8af6bf8a5" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39529 } } }
I20260927 18:47:39.546747 7136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:39.547082 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ea22788aab40d48b277db8af6bf8a5. Latest consensus state: current_term: 1 leader_uuid: "52ea22788aab40d48b277db8af6bf8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ea22788aab40d48b277db8af6bf8a5" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 39529 } } }
I20260927 18:47:39.547297 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:39.548314 7144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:39.550895 7144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:39.556828 7144 catalog_manager.cc:1347] Generated new cluster ID: 3640b72c707b499aa59ee6cb4ddc8724
I20260927 18:47:39.556905 7144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:39.586690 7144 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:39.587714 7144 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:39.596386 7144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52ea22788aab40d48b277db8af6bf8a5: Generated new TSK 0
I20260927 18:47:39.597151 7144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:47:39.608318 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39529
--builtin_ntp_servers=127.29.159.84:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 18:47:39.724345 7154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:39.724637 7154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:39.724676 7154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:39.728768 7154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:39.728829 7154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:39.728885 7154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:39.733951 7154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:39.735013 7154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:39.736389 7154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:39.746058 7159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:39.746070 7160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:39.746141 7162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:39.746380 7154 server_base.cc:1034] running on GCE node
I20260927 18:47:39.746809 7154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:39.747557 7154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:39.748729 7154 hybrid_clock.cc:648] HybridClock initialized: now 1790534859748703 us; error 26 us; skew 500 ppm
I20260927 18:47:39.750444 7154 webserver.cc:460] Webserver started at http://127.29.159.65:34479/ using document root <none> and password file <none>
I20260927 18:47:39.751017 7154 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:39.751078 7154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:39.751298 7154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:39.753326 7154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "0404ce26959245d6b889d01e42a43f6f"
format_stamp: "Formatted at 2026-09-27 18:47:39 on dist-test-slave-8tg2"
server_key: "33558d37d8a0744e0a273cdb4ee7b5af"
server_key_iv: "4f35746d3d037e6adbc137b9e028b008"
server_key_version: "encryptionkey@0"
I20260927 18:47:39.753891 7154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "0404ce26959245d6b889d01e42a43f6f"
format_stamp: "Formatted at 2026-09-27 18:47:39 on dist-test-slave-8tg2"
server_key: "33558d37d8a0744e0a273cdb4ee7b5af"
server_key_iv: "4f35746d3d037e6adbc137b9e028b008"
server_key_version: "encryptionkey@0"
I20260927 18:47:39.757468 7154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 18:47:39.759994 7168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:39.761238 7154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:39.761377 7154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "0404ce26959245d6b889d01e42a43f6f"
format_stamp: "Formatted at 2026-09-27 18:47:39 on dist-test-slave-8tg2"
server_key: "33558d37d8a0744e0a273cdb4ee7b5af"
server_key_iv: "4f35746d3d037e6adbc137b9e028b008"
server_key_version: "encryptionkey@0"
I20260927 18:47:39.761495 7154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:39.781294 7154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:39.782068 7154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:39.782286 7154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:39.782920 7154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:39.783959 7154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:39.784037 7154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:39.784113 7154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:39.784170 7154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:39.794226 7154 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:39943
I20260927 18:47:39.794245 7281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:39943 every 8 connection(s)
I20260927 18:47:39.795289 7154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:39.804982 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 7154
I20260927 18:47:39.805133 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:47:39.805130 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:39.797259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:35345 (local address 127.29.159.126:39529)
0927 18:47:39.797404 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:39.797408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:39.798584 (+ 1176us) server_negotiation.cc:407] Connection header received
0927 18:47:39.800078 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:39.800081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:39.800140 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:39.800223 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:39.801178 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.801740 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.802960 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.803195 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.803315 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:39.804588 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:39.804592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:39.804595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:39.804611 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:39.804638 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:39.804646 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:39.804728 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:39.804906 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 18:47:39.804980 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:47:39.805524 30333 external_mini_cluster.cc:1428] Setting key 197fa71df28a5e64200d16f164cd9f85
I20260927 18:47:39.806166 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:39.797656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39529 (local address 127.29.159.65:35345)
0927 18:47:39.798406 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:39.798451 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:39.799846 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:39.800402 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:39.800409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:39.800496 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:39.801007 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:39.801020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.801878 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.801882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:39.802804 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:39.802814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.803359 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.803363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:39.803609 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:39.804299 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:39.804300 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:39.804342 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:39.804409 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:39.804741 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:39.804753 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:39.804761 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:47:39.804901 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 18:47:39.805213 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260927 18:47:39.807487 7282 heartbeater.cc:347] Connected to a master server at 127.29.159.126:39529
I20260927 18:47:39.807752 7282 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:39.807829 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39529
--builtin_ntp_servers=127.29.159.84:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 18:47:39.808329 7282 heartbeater.cc:511] Master 127.29.159.126:39529 requested a full tablet report, sending...
I20260927 18:47:39.809868 7094 ts_manager.cc:194] Registered new tserver with Master: 0404ce26959245d6b889d01e42a43f6f (127.29.159.65:39943)
I20260927 18:47:39.811005 7094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.65:35345
W20260927 18:47:39.919564 7285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:39.919888 7285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:39.919960 7285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:39.924039 7285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:39.924134 7285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:39.924225 7285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:39.928978 7285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:39.930050 7285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:39.931511 7285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:39.941112 7290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:39.941098 7291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:39.941087 7293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:39.941303 7285 server_base.cc:1034] running on GCE node
I20260927 18:47:39.941811 7285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:39.942438 7285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:39.943598 7285 hybrid_clock.cc:648] HybridClock initialized: now 1790534859943566 us; error 45 us; skew 500 ppm
I20260927 18:47:39.945500 7285 webserver.cc:460] Webserver started at http://127.29.159.66:36457/ using document root <none> and password file <none>
I20260927 18:47:39.946105 7285 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:39.946171 7285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:39.946453 7285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:39.948664 7285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "9ea55541e4a34ed68cdd13bc0b8ccf62"
format_stamp: "Formatted at 2026-09-27 18:47:39 on dist-test-slave-8tg2"
server_key: "6d943f8f28dbece6123bfea7c1e0275d"
server_key_iv: "0b0d305e86f78d10f7cb7e49e773bf5a"
server_key_version: "encryptionkey@0"
I20260927 18:47:39.949240 7285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "9ea55541e4a34ed68cdd13bc0b8ccf62"
format_stamp: "Formatted at 2026-09-27 18:47:39 on dist-test-slave-8tg2"
server_key: "6d943f8f28dbece6123bfea7c1e0275d"
server_key_iv: "0b0d305e86f78d10f7cb7e49e773bf5a"
server_key_version: "encryptionkey@0"
I20260927 18:47:39.952992 7285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 18:47:39.955626 7299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:39.956851 7285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:39.957005 7285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "9ea55541e4a34ed68cdd13bc0b8ccf62"
format_stamp: "Formatted at 2026-09-27 18:47:39 on dist-test-slave-8tg2"
server_key: "6d943f8f28dbece6123bfea7c1e0275d"
server_key_iv: "0b0d305e86f78d10f7cb7e49e773bf5a"
server_key_version: "encryptionkey@0"
I20260927 18:47:39.957116 7285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:39.973174 7285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:39.973836 7285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:39.974072 7285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:39.974654 7285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:39.975627 7285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:39.975677 7285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:39.975752 7285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:39.975790 7285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:39.984614 7285 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:44023
I20260927 18:47:39.984655 7412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:44023 every 8 connection(s)
I20260927 18:47:39.985472 7285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:39.994119 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 7285
I20260927 18:47:39.994256 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:39.994624 30333 external_mini_cluster.cc:1428] Setting key 47be15a502f1c6cc3811d48debca0d77
I20260927 18:47:39.994864 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:39.987380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:40647 (local address 127.29.159.126:39529)
0927 18:47:39.987569 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:39.987573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:39.988613 (+ 1040us) server_negotiation.cc:407] Connection header received
0927 18:47:39.989872 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:39.989875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:39.989920 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:39.990002 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:39.990971 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.991566 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.992765 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.992987 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.993080 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:39.994280 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:39.994285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:39.994289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:39.994308 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:39.994353 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:39.994365 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:39.994436 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:39.994609 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 18:47:39.994718 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 18:47:39.995909 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:39.987793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39529 (local address 127.29.159.66:40647)
0927 18:47:39.988420 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:39.988458 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:39.989669 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:39.990166 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:39.990172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:39.990267 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:39.990756 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:39.990787 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.991699 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.991702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:39.992648 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:39.992657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:39.993125 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:39.993128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:39.993275 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:39.993927 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:39.993929 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:39.993984 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:39.994072 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:39.994446 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:39.994456 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:39.994465 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:47:39.994605 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 18:47:39.994898 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
I20260927 18:47:39.996838 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39529
--builtin_ntp_servers=127.29.159.84:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 18:47:39.997105 7413 heartbeater.cc:347] Connected to a master server at 127.29.159.126:39529
I20260927 18:47:39.997377 7413 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:39.997995 7413 heartbeater.cc:511] Master 127.29.159.126:39529 requested a full tablet report, sending...
I20260927 18:47:39.999178 7094 ts_manager.cc:194] Registered new tserver with Master: 9ea55541e4a34ed68cdd13bc0b8ccf62 (127.29.159.66:44023)
I20260927 18:47:39.999867 7094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.66:40647
W20260927 18:47:40.107625 7416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:40.107929 7416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:40.107969 7416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:40.112210 7416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:40.112280 7416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:40.112368 7416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:40.117377 7416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:39529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:40.118613 7416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:40.120137 7416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:40.130067 7422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:40.130177 7416 server_base.cc:1034] running on GCE node
W20260927 18:47:40.130084 7424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:40.130067 7421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:40.130887 7416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:40.131619 7416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:40.132851 7416 hybrid_clock.cc:648] HybridClock initialized: now 1790534860132820 us; error 51 us; skew 500 ppm
I20260927 18:47:40.134783 7416 webserver.cc:460] Webserver started at http://127.29.159.67:38121/ using document root <none> and password file <none>
I20260927 18:47:40.135376 7416 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:40.135500 7416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:40.135741 7416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:40.138007 7416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "ae78aae52b3743b4ab7cdc3dc50c3f07"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "3b803f2ba2abac27325f1e95a73b50f4"
server_key_iv: "5dddbb202c8472d7fed024ca0bc68484"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.138597 7416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "ae78aae52b3743b4ab7cdc3dc50c3f07"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "3b803f2ba2abac27325f1e95a73b50f4"
server_key_iv: "5dddbb202c8472d7fed024ca0bc68484"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.142328 7416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 18:47:40.145045 7430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:40.146454 7416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:47:40.146584 7416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "ae78aae52b3743b4ab7cdc3dc50c3f07"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "3b803f2ba2abac27325f1e95a73b50f4"
server_key_iv: "5dddbb202c8472d7fed024ca0bc68484"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.146714 7416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:40.171296 7416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:40.172115 7416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:40.172325 7416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:40.172935 7416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:40.173907 7416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:40.173964 7416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:40.174005 7416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:40.174026 7416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:40.182850 7416 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:38687
I20260927 18:47:40.182878 7543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:38687 every 8 connection(s)
I20260927 18:47:40.183869 7416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:40.193512 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 7416
I20260927 18:47:40.193638 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:40.193918 30333 external_mini_cluster.cc:1428] Setting key 11aa15018881860d187534bf8d117ade
I20260927 18:47:40.194078 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:40.185803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:57245 (local address 127.29.159.126:39529)
0927 18:47:40.185957 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:40.185960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:40.187137 (+ 1177us) server_negotiation.cc:407] Connection header received
0927 18:47:40.188706 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:40.188710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:40.188768 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:40.188894 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:40.189847 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:40.190388 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:40.191914 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:40.192171 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:40.192293 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:40.193457 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:40.193463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:40.193470 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:40.193484 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:40.193504 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:40.193512 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:40.193641 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:40.193857 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 18:47:40.193926 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:47:40.195225 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:40.186195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:39529 (local address 127.29.159.67:57245)
0927 18:47:40.186907 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:40.186950 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:40.188437 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:40.189079 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:40.189086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:40.189175 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:40.189695 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:40.189709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:40.190523 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:40.190526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:40.191774 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:40.191783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:40.192335 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:40.192339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:40.192526 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:40.193180 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:40.193182 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:40.193224 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:40.193289 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:40.193685 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:40.193695 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:40.193701 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:47:40.193856 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 18:47:40.194170 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":116,"threads_started":1}
I20260927 18:47:40.196362 7544 heartbeater.cc:347] Connected to a master server at 127.29.159.126:39529
I20260927 18:47:40.196589 7544 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:40.197078 7544 heartbeater.cc:511] Master 127.29.159.126:39529 requested a full tablet report, sending...
I20260927 18:47:40.198123 7093 ts_manager.cc:194] Registered new tserver with Master: ae78aae52b3743b4ab7cdc3dc50c3f07 (127.29.159.67:38687)
I20260927 18:47:40.198673 7093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.67:57245
I20260927 18:47:40.207854 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:47:40.210829 7133 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:40.209790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.29.159.126:39529)
0927 18:47:40.209976 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:40.209981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:40.210123 (+ 142us) server_negotiation.cc:407] Connection header received
0927 18:47:40.210293 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:40.210296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:40.210347 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:40.210435 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:40.210701 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50100: BlockingRecv error: recv got EOF from 127.0.0.1:50100 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 18:47:40.211293 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 7154
I20260927 18:47:40.217530 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 7285
I20260927 18:47:40.222678 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 7416
I20260927 18:47:40.227689 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 7063
2026-09-27T18:47:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (916 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T18:47:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:40Z Disabled control of system clock
I20260927 18:47:40.256138 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44379
--webserver_interface=127.29.159.126
--webserver_port=0
--builtin_ntp_servers=127.29.159.84:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:47:40.365566 7555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:40.365825 7555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:40.365861 7555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:40.369874 7555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:40.369925 7555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:40.369941 7555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:40.369958 7555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:40.369972 7555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:40.375030 7555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:44379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:40.376219 7555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:40.377599 7555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:40.386161 7561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:40.386219 7563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:40.386237 7560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:40.386462 7555 server_base.cc:1034] running on GCE node
I20260927 18:47:40.387212 7555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:40.388458 7555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:40.389660 7555 hybrid_clock.cc:648] HybridClock initialized: now 1790534860389633 us; error 47 us; skew 500 ppm
I20260927 18:47:40.391577 7555 webserver.cc:460] Webserver started at http://127.29.159.126:33345/ using document root <none> and password file <none>
I20260927 18:47:40.392300 7555 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:40.392396 7555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:40.392609 7555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:40.394639 7555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "f70107b6059e4b4c8f997026538056aa"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "c111114d99f160d3f10b2a11c0e2df42"
server_key_iv: "ec61ace023e848cd858db00981dbda80"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.395220 7555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "f70107b6059e4b4c8f997026538056aa"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "c111114d99f160d3f10b2a11c0e2df42"
server_key_iv: "ec61ace023e848cd858db00981dbda80"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.398926 7555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 18:47:40.401494 7569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:40.402801 7555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:47:40.402956 7555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "f70107b6059e4b4c8f997026538056aa"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "c111114d99f160d3f10b2a11c0e2df42"
server_key_iv: "ec61ace023e848cd858db00981dbda80"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.403080 7555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:40.407658 7555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:40.408277 7555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:40.408459 7555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:40.415246 7555 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:44379
I20260927 18:47:40.415267 7621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:44379 every 8 connection(s)
I20260927 18:47:40.416505 7555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:40.419754 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:40.421676 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 7555
I20260927 18:47:40.421797 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:40.422079 30333 external_mini_cluster.cc:1428] Setting key eb3b3b67b3db4af9db21003beac8f568
I20260927 18:47:40.426545 7622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa: Bootstrap starting.
I20260927 18:47:40.427912 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:40.423548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.29.159.126:44379)
0927 18:47:40.424227 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:40.424243 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:40.424279 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:47:40.424911 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:40.424927 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:40.425286 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:40.425622 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:40.426474 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:40.426479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:40.426491 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:40.426501 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:40.426559 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:40.426574 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:40.426747 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:40.426856 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 18:47:40.427162 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260927 18:47:40.428937 7622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:40.429708 7622 log.cc:824] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:40.431609 7622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa: No bootstrap required, opened a new log
I20260927 18:47:40.434993 7622 raft_consensus.cc:348] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70107b6059e4b4c8f997026538056aa" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44379 } }
I20260927 18:47:40.435215 7622 raft_consensus.cc:374] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:40.435298 7622 raft_consensus.cc:729] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f70107b6059e4b4c8f997026538056aa, State: Initialized, Role: FOLLOWER
I20260927 18:47:40.435783 7622 consensus_queue.cc:260] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70107b6059e4b4c8f997026538056aa" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44379 } }
I20260927 18:47:40.435930 7622 raft_consensus.cc:388] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:40.435993 7622 raft_consensus.cc:482] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:40.436088 7622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:40.436975 7622 raft_consensus.cc:504] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70107b6059e4b4c8f997026538056aa" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44379 } }
I20260927 18:47:40.437315 7622 leader_election.cc:302] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f70107b6059e4b4c8f997026538056aa; no voters:
I20260927 18:47:40.437559 7622 leader_election.cc:288] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:40.437733 7627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:40.437990 7627 raft_consensus.cc:686] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [term 1 LEADER]: Becoming Leader. State: Replica: f70107b6059e4b4c8f997026538056aa, State: Running, Role: LEADER
I20260927 18:47:40.438359 7627 consensus_queue.cc:237] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70107b6059e4b4c8f997026538056aa" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44379 } }
I20260927 18:47:40.438827 7622 sys_catalog.cc:564] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:40.440101 7629 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f70107b6059e4b4c8f997026538056aa. Latest consensus state: current_term: 1 leader_uuid: "f70107b6059e4b4c8f997026538056aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70107b6059e4b4c8f997026538056aa" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44379 } } }
I20260927 18:47:40.440147 7628 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f70107b6059e4b4c8f997026538056aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70107b6059e4b4c8f997026538056aa" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44379 } } }
I20260927 18:47:40.440214 7629 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:40.440237 7628 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:40.440587 7636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:40.443809 7636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:40.449119 7636 catalog_manager.cc:1347] Generated new cluster ID: 1cfc5b77a1cb4c328ebacb652a38cadb
I20260927 18:47:40.449191 7636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:40.462395 7636 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:40.462702 7636 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:40.476557 7636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f70107b6059e4b4c8f997026538056aa: Generated new TSK 0
I20260927 18:47:40.477368 7636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:47:40.484949 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44379
--builtin_ntp_servers=127.29.159.84:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:47:40.613845 7646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:40.614122 7646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:40.614163 7646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:40.618230 7646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:40.618299 7646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:40.618358 7646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:40.623190 7646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:40.624341 7646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:40.625654 7646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:40.635330 7651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:40.635330 7652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:40.635360 7654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:40.635624 7646 server_base.cc:1034] running on GCE node
I20260927 18:47:40.636026 7646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:40.636617 7646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:40.637778 7646 hybrid_clock.cc:648] HybridClock initialized: now 1790534860637767 us; error 26 us; skew 500 ppm
I20260927 18:47:40.639752 7646 webserver.cc:460] Webserver started at http://127.29.159.65:40539/ using document root <none> and password file <none>
I20260927 18:47:40.640362 7646 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:40.640429 7646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:40.640652 7646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:40.642657 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "891b7e7f34f04a2c86ba43070392da46"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "4359847130cfe8c92736c3c27833c154"
server_key_iv: "08e53b2ead458bd93c9ecd3dd638ee3e"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.643224 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "891b7e7f34f04a2c86ba43070392da46"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "4359847130cfe8c92736c3c27833c154"
server_key_iv: "08e53b2ead458bd93c9ecd3dd638ee3e"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.646739 7646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:47:40.649343 7660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:40.650532 7646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:40.650664 7646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "891b7e7f34f04a2c86ba43070392da46"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "4359847130cfe8c92736c3c27833c154"
server_key_iv: "08e53b2ead458bd93c9ecd3dd638ee3e"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.650779 7646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:40.655184 7646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:40.655813 7646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:40.655997 7646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:40.656507 7646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:40.657359 7646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:40.657430 7646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:40.657493 7646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:40.657543 7646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:40.665926 7646 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:43799
I20260927 18:47:40.665958 7773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:43799 every 8 connection(s)
I20260927 18:47:40.666761 7646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:47:40.671999 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 7646
I20260927 18:47:40.672127 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:47:40.672425 30333 external_mini_cluster.cc:1428] Setting key 6973ae5b1ae5c2e30d1ce9e85219eb7e
I20260927 18:47:40.672628 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:40.668496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:35875 (local address 127.29.159.126:44379)
0927 18:47:40.668640 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:40.668645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:40.669663 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 18:47:40.670934 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:40.670937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:40.670973 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:40.671045 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:40.672147 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:40.672151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:40.672154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:40.672171 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:40.672194 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:40.672207 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:40.672321 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:40.672459 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 18:47:40.672503 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:47:40.673568 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:40.668902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44379 (local address 127.29.159.65:35875)
0927 18:47:40.669514 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:40.669553 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:40.670758 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:40.671163 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:40.671170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:40.671257 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:40.671830 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:40.671833 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:40.671888 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:40.671977 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:40.672304 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:40.672313 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:40.672319 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:47:40.672501 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 18:47:40.672724 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260927 18:47:40.674362 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44379
--builtin_ntp_servers=127.29.159.84:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 18:47:40.674708 7774 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44379
I20260927 18:47:40.674935 7774 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:40.675372 7774 heartbeater.cc:511] Master 127.29.159.126:44379 requested a full tablet report, sending...
I20260927 18:47:40.676887 7586 ts_manager.cc:194] Registered new tserver with Master: 891b7e7f34f04a2c86ba43070392da46 (127.29.159.65:43799)
W20260927 18:47:40.793102 7777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:40.793401 7777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:40.793486 7777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:40.799400 7777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:40.799495 7777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:40.799590 7777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:40.804833 7777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:40.806026 7777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:40.807802 7777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:40.820827 7783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:40.820842 7785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:40.820827 7782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:40.821324 7777 server_base.cc:1034] running on GCE node
I20260927 18:47:40.821722 7777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:40.822392 7777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:40.823567 7777 hybrid_clock.cc:648] HybridClock initialized: now 1790534860823537 us; error 50 us; skew 500 ppm
I20260927 18:47:40.825968 7777 webserver.cc:460] Webserver started at http://127.29.159.66:42613/ using document root <none> and password file <none>
I20260927 18:47:40.826593 7777 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:40.826684 7777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:40.826897 7777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:40.829706 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "5371d4a1c3724e8d88d569ad09fea95f"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "4ab7f8b0b05a421f2a4a0de449766d43"
server_key_iv: "03dbeee3a0ca177b9762fc042ca8b89c"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.830421 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "5371d4a1c3724e8d88d569ad09fea95f"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "4ab7f8b0b05a421f2a4a0de449766d43"
server_key_iv: "03dbeee3a0ca177b9762fc042ca8b89c"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.834280 7777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 18:47:40.836827 7791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:40.838078 7777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:47:40.838222 7777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "5371d4a1c3724e8d88d569ad09fea95f"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "4ab7f8b0b05a421f2a4a0de449766d43"
server_key_iv: "03dbeee3a0ca177b9762fc042ca8b89c"
server_key_version: "encryptionkey@0"
I20260927 18:47:40.838338 7777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:40.842903 7777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:40.843538 7777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:40.843707 7777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:40.844218 7777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:40.845050 7777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:40.845119 7777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:40.845182 7777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:40.845242 7777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:40.853721 7777 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:41347
I20260927 18:47:40.853749 7904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:41347 every 8 connection(s)
I20260927 18:47:40.854624 7777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:40.860903 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 7777
I20260927 18:47:40.860942 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:40.856672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:54477 (local address 127.29.159.126:44379)
0927 18:47:40.856817 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:40.856821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:40.857778 (+ 957us) server_negotiation.cc:407] Connection header received
0927 18:47:40.859124 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:40.859127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:40.859174 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:40.859288 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:40.860470 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:40.860474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:40.860476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:40.860488 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:40.860509 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:40.860517 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:40.860624 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:40.860799 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 18:47:40.860838 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:47:40.861033 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:40.861328 30333 external_mini_cluster.cc:1428] Setting key 609dd29a9a706835006027ce635c4769
I20260927 18:47:40.862010 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:40.857004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44379 (local address 127.29.159.66:54477)
0927 18:47:40.857623 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:40.857662 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:40.858900 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:40.859506 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:40.859520 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:40.859621 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:40.860218 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:40.860220 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:40.860260 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:40.860325 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:40.860617 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:40.860625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:40.860632 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:47:40.860815 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 18:47:40.861069 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":113,"threads_started":1}
I20260927 18:47:40.863126 7905 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44379
I20260927 18:47:40.863354 7905 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:40.863513 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44379
--builtin_ntp_servers=127.29.159.84:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 18:47:40.863839 7905 heartbeater.cc:511] Master 127.29.159.126:44379 requested a full tablet report, sending...
I20260927 18:47:40.864960 7585 ts_manager.cc:194] Registered new tserver with Master: 5371d4a1c3724e8d88d569ad09fea95f (127.29.159.66:41347)
W20260927 18:47:40.979673 7908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:40.979943 7908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:40.979983 7908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:40.984218 7908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:40.984273 7908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:40.984328 7908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:40.989008 7908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:40.990008 7908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:40.991349 7908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:41.001364 7914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:41.001436 7908 server_base.cc:1034] running on GCE node
W20260927 18:47:41.001364 7913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:41.001394 7916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:41.002137 7908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:41.002825 7908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:41.004029 7908 hybrid_clock.cc:648] HybridClock initialized: now 1790534861004015 us; error 27 us; skew 500 ppm
I20260927 18:47:41.006017 7908 webserver.cc:460] Webserver started at http://127.29.159.67:35413/ using document root <none> and password file <none>
I20260927 18:47:41.006778 7908 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:41.006873 7908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:41.007103 7908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:41.009330 7908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "1adfa1fdfa394e7ea9bed6fcd97cf756"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "d5a0f08637ca1a58c607d8e5e6eabf37"
server_key_iv: "0fabd4a7089812d71e84e77eece6c508"
server_key_version: "encryptionkey@0"
I20260927 18:47:41.009934 7908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "1adfa1fdfa394e7ea9bed6fcd97cf756"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "d5a0f08637ca1a58c607d8e5e6eabf37"
server_key_iv: "0fabd4a7089812d71e84e77eece6c508"
server_key_version: "encryptionkey@0"
I20260927 18:47:41.013870 7908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:47:41.016554 7922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:41.017827 7908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:47:41.017982 7908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "1adfa1fdfa394e7ea9bed6fcd97cf756"
format_stamp: "Formatted at 2026-09-27 18:47:40 on dist-test-slave-8tg2"
server_key: "d5a0f08637ca1a58c607d8e5e6eabf37"
server_key_iv: "0fabd4a7089812d71e84e77eece6c508"
server_key_version: "encryptionkey@0"
I20260927 18:47:41.018129 7908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:41.022890 7908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:41.023550 7908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:41.023733 7908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:41.024279 7908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:41.025189 7908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:41.025261 7908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:41.025323 7908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:41.025372 7908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:41.034327 7908 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:46253
I20260927 18:47:41.034344 8035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:46253 every 8 connection(s)
I20260927 18:47:41.035221 7908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:41.039968 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 7908
I20260927 18:47:41.040081 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:41.040436 30333 external_mini_cluster.cc:1428] Setting key ff8adaac1de03072ec2df2cfccc0951d
I20260927 18:47:41.042047 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:41.037121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:38819 (local address 127.29.159.126:44379)
0927 18:47:41.037290 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:41.037295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:41.038434 (+ 1139us) server_negotiation.cc:407] Connection header received
0927 18:47:41.040040 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:41.040046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:41.040095 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:41.040272 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:41.041554 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:41.041558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:41.041561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:47:41.041574 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:41.041595 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:47:41.041603 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:41.041710 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:41.041876 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 18:47:41.041922 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:47:41.042959 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:41.037522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44379 (local address 127.29.159.67:38819)
0927 18:47:41.038168 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:41.038210 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:41.039814 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:41.040540 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:41.040550 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:41.040658 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:41.041288 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:47:41.041290 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:41.041332 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:47:41.041402 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:41.041696 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:41.041704 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:41.041710 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:47:41.041907 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 18:47:41.042228 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":115,"threads_started":1}
I20260927 18:47:41.043989 8036 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44379
I20260927 18:47:41.044183 8036 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:41.044528 8036 heartbeater.cc:511] Master 127.29.159.126:44379 requested a full tablet report, sending...
I20260927 18:47:41.045485 7585 ts_manager.cc:194] Registered new tserver with Master: 1adfa1fdfa394e7ea9bed6fcd97cf756 (127.29.159.67:46253)
I20260927 18:47:41.054350 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:47:41.057392 7625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:47:41.056488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.29.159.126:44379)
0927 18:47:41.056633 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:41.056637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:41.056819 (+ 182us) server_negotiation.cc:407] Connection header received
0927 18:47:41.056902 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:41.056904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:41.056937 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:41.057004 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:41.057262 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36194: BlockingRecv error: recv got EOF from 127.0.0.1:36194 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:47:41.057708 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 7646
I20260927 18:47:41.063059 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 7777
I20260927 18:47:41.068266 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 7908
I20260927 18:47:41.073333 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 7555
2026-09-27T18:47:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (845 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:47:41 dist-test-slave-8tg2 krb5kdc[8046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:47:41 dist-test-slave-8tg2 krb5kdc[8046](info): set up 2 sockets
Sep 27 18:47:41 dist-test-slave-8tg2 krb5kdc[8046](info): commencing operation
krb5kdc: starting...
W20260927 18:47:42.148584 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:47:42 dist-test-slave-8tg2 krb5kdc[8046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:47:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:47:42.272307 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44021
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:44021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:42.382602 8059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:42.382849 8059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:42.382896 8059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:42.386862 8059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:47:42.386921 8059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:42.386940 8059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:42.386956 8059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:47:42.386971 8059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:47:42.392257 8059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:44021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:42.393245 8059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:42.394213 8059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:42.402812 8065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:42.402798 8067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:42.402814 8064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:42.403005 8059 server_base.cc:1034] running on GCE node
I20260927 18:47:42.403803 8059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:42.404908 8059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:42.406091 8059 hybrid_clock.cc:648] HybridClock initialized: now 1790534862406083 us; error 36 us; skew 500 ppm
Sep 27 18:47:42 dist-test-slave-8tg2 krb5kdc[8046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534862, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:42.408869 8059 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:47:42.409737 8059 webserver.cc:460] Webserver started at http://127.29.159.126:33217/ using document root <none> and password file <none>
I20260927 18:47:42.410286 8059 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:42.410387 8059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:42.410611 8059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:42.412679 8059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "b0580fdd11724e03a76a2f4ce4bf0a76"
format_stamp: "Formatted at 2026-09-27 18:47:42 on dist-test-slave-8tg2"
server_key: "3162c48396a28714e314e6442df79822"
server_key_iv: "04cbe40c263dd0433ddee58af85a6ead"
server_key_version: "encryptionkey@0"
I20260927 18:47:42.413260 8059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "b0580fdd11724e03a76a2f4ce4bf0a76"
format_stamp: "Formatted at 2026-09-27 18:47:42 on dist-test-slave-8tg2"
server_key: "3162c48396a28714e314e6442df79822"
server_key_iv: "04cbe40c263dd0433ddee58af85a6ead"
server_key_version: "encryptionkey@0"
I20260927 18:47:42.417075 8059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 18:47:42.419567 8074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:42.420845 8059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:47:42.420991 8059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "b0580fdd11724e03a76a2f4ce4bf0a76"
format_stamp: "Formatted at 2026-09-27 18:47:42 on dist-test-slave-8tg2"
server_key: "3162c48396a28714e314e6442df79822"
server_key_iv: "04cbe40c263dd0433ddee58af85a6ead"
server_key_version: "encryptionkey@0"
I20260927 18:47:42.421104 8059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:42.444538 8059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:42.448005 8059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:42.448230 8059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:42.455499 8059 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:44021
I20260927 18:47:42.455497 8126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:44021 every 8 connection(s)
I20260927 18:47:42.456492 8059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:47:42.458458 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 8059
I20260927 18:47:42.458590 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:47:42.458899 30333 external_mini_cluster.cc:1428] Setting key 1b48eea9bc88ad3ec93ecc6e07ddb208
I20260927 18:47:42.459692 8127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:42.466230 8127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76: Bootstrap starting.
Sep 27 18:47:42 dist-test-slave-8tg2 krb5kdc[8046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:42.468803 8127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:42.469504 8127 log.cc:824] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:42.471395 8127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76: No bootstrap required, opened a new log
I20260927 18:47:42.473268 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:42.460246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.29.159.126:44021)
0927 18:47:42.460629 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:42.460651 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:42.460686 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:47:42.461325 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:42.461341 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:42.461656 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:42.461935 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:42.462873 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:42.463977 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:42.465153 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:42.465628 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:42.465753 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:42.468480 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:42.468485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:42.468497 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:42.468524 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:42.470503 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:42.470941 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:42.470947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:42.470959 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:42.471071 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:42.471364 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:42.471368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:42.471370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:42.471832 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:42.472028 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:42.472324 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 18:47:42.472599 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":134,"threads_started":1}
I20260927 18:47:42.474953 8127 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0580fdd11724e03a76a2f4ce4bf0a76" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44021 } }
I20260927 18:47:42.475193 8127 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:42.475302 8127 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0580fdd11724e03a76a2f4ce4bf0a76, State: Initialized, Role: FOLLOWER
I20260927 18:47:42.475745 8127 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0580fdd11724e03a76a2f4ce4bf0a76" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44021 } }
I20260927 18:47:42.475896 8127 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:47:42.475967 8127 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:47:42.476097 8127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:42.476930 8127 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0580fdd11724e03a76a2f4ce4bf0a76" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44021 } }
I20260927 18:47:42.477267 8127 leader_election.cc:302] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0580fdd11724e03a76a2f4ce4bf0a76; no voters:
I20260927 18:47:42.477555 8127 leader_election.cc:288] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:47:42.477701 8132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:42.477960 8132 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [term 1 LEADER]: Becoming Leader. State: Replica: b0580fdd11724e03a76a2f4ce4bf0a76, State: Running, Role: LEADER
I20260927 18:47:42.478257 8132 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0580fdd11724e03a76a2f4ce4bf0a76" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44021 } }
I20260927 18:47:42.478758 8127 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:47:42.479897 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0580fdd11724e03a76a2f4ce4bf0a76. Latest consensus state: current_term: 1 leader_uuid: "b0580fdd11724e03a76a2f4ce4bf0a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0580fdd11724e03a76a2f4ce4bf0a76" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44021 } } }
I20260927 18:47:42.480026 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:42.480406 8141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:47:42.480664 8133 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0580fdd11724e03a76a2f4ce4bf0a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0580fdd11724e03a76a2f4ce4bf0a76" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44021 } } }
I20260927 18:47:42.480841 8133 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76 [sys.catalog]: This master's current role is: LEADER
I20260927 18:47:42.482846 8141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:47:42.487856 8141 catalog_manager.cc:1347] Generated new cluster ID: d6e932807b484ff684f725cdccd5e187
I20260927 18:47:42.487984 8141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:47:42.512953 8141 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:47:42.513912 8141 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:47:42.517678 8141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76: Generated new TSK 0
I20260927 18:47:42.518433 8141 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:47:42.550241 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44021
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:42.684492 8153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:42.684772 8153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:42.684815 8153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:42.688920 8153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:42.688982 8153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:42.689046 8153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:47:42.694211 8153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:42.695240 8153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:42.696290 8153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:42.706758 8158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:42.706763 8159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:42.706714 8161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:42.706964 8153 server_base.cc:1034] running on GCE node
I20260927 18:47:42.707532 8153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:42.708225 8153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:42.709414 8153 hybrid_clock.cc:648] HybridClock initialized: now 1790534862709400 us; error 34 us; skew 500 ppm
Sep 27 18:47:42 dist-test-slave-8tg2 krb5kdc[8046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534862, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:42.712199 8153 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:47:42.713125 8153 webserver.cc:460] Webserver started at http://127.29.159.65:41739/ using document root <none> and password file <none>
I20260927 18:47:42.713737 8153 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:42.713827 8153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:42.714058 8153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:42.716346 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "6ab0d2f6f392449ea44e3820399b204b"
format_stamp: "Formatted at 2026-09-27 18:47:42 on dist-test-slave-8tg2"
server_key: "c4e7f358ed874d10a1a78a21ba376301"
server_key_iv: "a3f3700e5067133af6c0f3610355ead4"
server_key_version: "encryptionkey@0"
I20260927 18:47:42.717080 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "6ab0d2f6f392449ea44e3820399b204b"
format_stamp: "Formatted at 2026-09-27 18:47:42 on dist-test-slave-8tg2"
server_key: "c4e7f358ed874d10a1a78a21ba376301"
server_key_iv: "a3f3700e5067133af6c0f3610355ead4"
server_key_version: "encryptionkey@0"
I20260927 18:47:42.722432 8153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:47:42.725405 8168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:42.726645 8153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 18:47:42.726775 8153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "6ab0d2f6f392449ea44e3820399b204b"
format_stamp: "Formatted at 2026-09-27 18:47:42 on dist-test-slave-8tg2"
server_key: "c4e7f358ed874d10a1a78a21ba376301"
server_key_iv: "a3f3700e5067133af6c0f3610355ead4"
server_key_version: "encryptionkey@0"
I20260927 18:47:42.726892 8153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:42.743755 8153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:42.746387 8153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:42.746575 8153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:42.747149 8153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:42.747996 8153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:42.748042 8153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:42.748088 8153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:42.748111 8153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:42.756558 8153 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:44213
I20260927 18:47:42.756597 8281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:44213 every 8 connection(s)
I20260927 18:47:42.757447 8153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
Sep 27 18:47:42 dist-test-slave-8tg2 krb5kdc[8046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534862, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:42.767747 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 8153
I20260927 18:47:42.767876 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:47:42.768148 30333 external_mini_cluster.cc:1428] Setting key eecdd972c7ad673a8b8da00b901d492b
I20260927 18:47:42.771533 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:42.759128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:38711 (local address 127.29.159.126:44021)
0927 18:47:42.759278 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:42.759281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:42.760133 (+ 852us) server_negotiation.cc:407] Connection header received
0927 18:47:42.761503 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:42.761506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:42.761566 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:42.761669 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:42.762818 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:42.763351 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:42.764555 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:42.764801 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:42.764916 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:42.767347 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:42.767350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:42.767352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:42.767376 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:42.769202 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:42.769809 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:42.769812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:42.769813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:42.769886 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:42.770273 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:42.770275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:42.770277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:42.770439 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:42.770516 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:42.771182 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 18:47:42.771280 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:47:42.772358 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:42.759572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44021 (local address 127.29.159.65:38711)
0927 18:47:42.759987 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:42.760026 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:42.761289 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:42.761854 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:42.761861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:42.762196 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:42.762665 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:42.762678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:42.763509 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:42.763517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:42.764396 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:42.764405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:42.764950 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:42.764954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:42.765120 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:42.765719 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:42.765735 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:42.767188 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:42.769322 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:42.769327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:42.769339 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:42.769714 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:42.769980 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:42.769983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:42.769985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:42.770169 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:42.770545 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:42.770555 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:42.770898 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 18:47:42.771125 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 18:47:42.771509 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260927 18:47:42.773617 8282 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44021
I20260927 18:47:42.773855 8282 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:42.774341 8282 heartbeater.cc:511] Master 127.29.159.126:44021 requested a full tablet report, sending...
I20260927 18:47:42.775916 8091 ts_manager.cc:194] Registered new tserver with Master: 6ab0d2f6f392449ea44e3820399b204b (127.29.159.65:44213)
I20260927 18:47:42.777024 8091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:38711
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:47:42.796626 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44021
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:42.907186 8287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:42.907492 8287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:42.907549 8287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:42.911497 8287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:42.911548 8287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:42.911610 8287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:47:42.916702 8287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:42.917721 8287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:42.918704 8287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:47:42.928314 8287 server_base.cc:1034] running on GCE node
W20260927 18:47:42.928277 8293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:42.928237 8292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:42.928236 8295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:42.928898 8287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:42.929602 8287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:42.930790 8287 hybrid_clock.cc:648] HybridClock initialized: now 1790534862930774 us; error 26 us; skew 500 ppm
Sep 27 18:47:42 dist-test-slave-8tg2 krb5kdc[8046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534862, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:42.933622 8287 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:47:42.934470 8287 webserver.cc:460] Webserver started at http://127.29.159.66:39351/ using document root <none> and password file <none>
I20260927 18:47:42.935047 8287 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:42.935141 8287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:42.935590 8287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:42.937685 8287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "fc3cf1fe7c51477aaa86cdbb545134a5"
format_stamp: "Formatted at 2026-09-27 18:47:42 on dist-test-slave-8tg2"
server_key: "705ec17df61ab453ec690dbb962055a1"
server_key_iv: "6aaf3a3d2691970c8f0cae4543b357cd"
server_key_version: "encryptionkey@0"
I20260927 18:47:42.938272 8287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "fc3cf1fe7c51477aaa86cdbb545134a5"
format_stamp: "Formatted at 2026-09-27 18:47:42 on dist-test-slave-8tg2"
server_key: "705ec17df61ab453ec690dbb962055a1"
server_key_iv: "6aaf3a3d2691970c8f0cae4543b357cd"
server_key_version: "encryptionkey@0"
I20260927 18:47:42.942076 8287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 18:47:42.944607 8302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:42.945883 8287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:47:42.946110 8287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "fc3cf1fe7c51477aaa86cdbb545134a5"
format_stamp: "Formatted at 2026-09-27 18:47:42 on dist-test-slave-8tg2"
server_key: "705ec17df61ab453ec690dbb962055a1"
server_key_iv: "6aaf3a3d2691970c8f0cae4543b357cd"
server_key_version: "encryptionkey@0"
I20260927 18:47:42.946225 8287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:42.972695 8287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:42.975870 8287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:42.976081 8287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:42.976677 8287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:42.977541 8287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:42.977587 8287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:42.977674 8287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:42.977715 8287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:42.987191 8287 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:45623
I20260927 18:47:42.987210 8415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:45623 every 8 connection(s)
I20260927 18:47:42.988153 8287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:47:42.993310 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 8287
I20260927 18:47:42.993445 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:47:42.993732 30333 external_mini_cluster.cc:1428] Setting key 5a74eb57dc309e79c6432791bc0a7f8b
Sep 27 18:47:42 dist-test-slave-8tg2 krb5kdc[8046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534862, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:43.003492 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:42.990135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:33593 (local address 127.29.159.126:44021)
0927 18:47:42.990296 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:42.990301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:42.991048 (+ 747us) server_negotiation.cc:407] Connection header received
0927 18:47:42.992507 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:42.992510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:42.992556 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:42.992634 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:42.994024 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:42.994673 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:42.996027 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:42.996256 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:42.996364 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:42.999173 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:42.999180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:42.999187 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:42.999221 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:43.001148 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:43.001688 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:43.001695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:43.001699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:43.001792 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:43.002184 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:43.002189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:43.002193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:43.002451 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:43.002571 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:43.003126 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 18:47:43.003274 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:47:43.004407 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:42.990455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44021 (local address 127.29.159.66:33593)
0927 18:47:42.990898 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:42.990939 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:42.992304 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:42.992806 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:42.992813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:42.993195 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:42.993862 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:42.993879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:42.994803 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:42.994807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:42.995893 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:42.995902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:42.996415 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:42.996419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:42.996613 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:42.997225 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:42.997240 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:42.998953 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:43.001289 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:43.001294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:43.001312 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:43.001577 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:43.001894 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:43.001897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:43.001898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:43.002032 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:43.002587 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:43.002599 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:43.002858 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 18:47:43.003031 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 18:47:43.003387 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260927 18:47:43.005798 8416 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44021
I20260927 18:47:43.006063 8416 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:47:43.006740 8416 heartbeater.cc:511] Master 127.29.159.126:44021 requested a full tablet report, sending...
I20260927 18:47:43.007915 8091 ts_manager.cc:194] Registered new tserver with Master: fc3cf1fe7c51477aaa86cdbb545134a5 (127.29.159.66:45623)
I20260927 18:47:43.008776 8091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:33593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:47:43.024022 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44021
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:47:43.134377 8421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:47:43.134663 8421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:47:43.134745 8421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:47:43.138911 8421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:47:43.138996 8421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:47:43.139091 8421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:47:43.144408 8421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:47:43.145519 8421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:47:43.146569 8421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:47:43.156162 8426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:43.156172 8429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:47:43.156167 8427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:47:43.156483 8421 server_base.cc:1034] running on GCE node
I20260927 18:47:43.156864 8421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:47:43.157492 8421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:47:43.158677 8421 hybrid_clock.cc:648] HybridClock initialized: now 1790534863158663 us; error 27 us; skew 500 ppm
Sep 27 18:47:43 dist-test-slave-8tg2 krb5kdc[8046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534863, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:47:43.161487 8421 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:47:43.162304 8421 webserver.cc:460] Webserver started at http://127.29.159.67:40165/ using document root <none> and password file <none>
I20260927 18:47:43.162848 8421 fs_manager.cc:359] Metadata directory not provided
I20260927 18:47:43.162945 8421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:47:43.163192 8421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:47:43.165247 8421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "e41e35b308f44384a382e29ad34bc838"
format_stamp: "Formatted at 2026-09-27 18:47:43 on dist-test-slave-8tg2"
server_key: "d8afbbf61c8c0990338e2e653a005de0"
server_key_iv: "597aa362f1b43af5b874fab1c80798ab"
server_key_version: "encryptionkey@0"
I20260927 18:47:43.165791 8421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "e41e35b308f44384a382e29ad34bc838"
format_stamp: "Formatted at 2026-09-27 18:47:43 on dist-test-slave-8tg2"
server_key: "d8afbbf61c8c0990338e2e653a005de0"
server_key_iv: "597aa362f1b43af5b874fab1c80798ab"
server_key_version: "encryptionkey@0"
I20260927 18:47:43.169358 8421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:47:43.171761 8436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:43.172921 8421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:47:43.173070 8421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "e41e35b308f44384a382e29ad34bc838"
format_stamp: "Formatted at 2026-09-27 18:47:43 on dist-test-slave-8tg2"
server_key: "d8afbbf61c8c0990338e2e653a005de0"
server_key_iv: "597aa362f1b43af5b874fab1c80798ab"
server_key_version: "encryptionkey@0"
I20260927 18:47:43.173180 8421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:47:43.193828 8421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:47:43.196762 8421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:47:43.196966 8421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:47:43.197530 8421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:47:43.198443 8421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:47:43.198513 8421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:43.198581 8421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:47:43.198632 8421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:47:43.207996 8421 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:34061
I20260927 18:47:43.208055 8549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:34061 every 8 connection(s)
I20260927 18:47:43.208911 8421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:47:43.210484 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 8421
I20260927 18:47:43.210626 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:47:43.210999 30333 external_mini_cluster.cc:1428] Setting key f28591dc36a623ba19a4044f102a77ca
Sep 27 18:47:43 dist-test-slave-8tg2 krb5kdc[8046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534863, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:47:43.222326 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.210792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:36575 (local address 127.29.159.126:44021)
0927 18:47:43.210958 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:43.210963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:43.211894 (+ 931us) server_negotiation.cc:407] Connection header received
0927 18:47:43.213503 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:43.213505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:43.213550 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:43.213620 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:43.214693 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.215254 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.216400 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.216614 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.216707 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.219104 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:43.219107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:43.219109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:43.219132 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:43.220471 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:43.220961 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:43.220963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:43.220965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:43.221051 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:43.221441 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:43.221443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:43.221445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:43.221607 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:43.221681 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:43.222117 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 18:47:43.222211 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:47:43.223161 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.211221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44021 (local address 127.29.159.67:36575)
0927 18:47:43.211742 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:43.211796 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:43.213319 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:43.213750 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:43.213756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:43.214092 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:47:43.214565 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.214577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.215360 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.215363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.216290 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.216299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.216747 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.216749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.216882 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.217450 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:47:43.217465 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:47:43.218961 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:47:43.220576 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:43.220582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:43.220594 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:43.220848 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:43.221161 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:47:43.221164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:47:43.221166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:47:43.221347 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:47:43.221698 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:47:43.221703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:47:43.221919 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 18:47:43.222072 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 18:47:43.222321 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":140,"threads_started":1}
I20260927 18:47:43.224496 8550 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44021
I20260927 18:47:43.224740 8550 heartbeater.cc:464] Registering TS with master...
I20260927 18:47:43.225274 8550 heartbeater.cc:511] Master 127.29.159.126:44021 requested a full tablet report, sending...
I20260927 18:47:43.226155 8090 ts_manager.cc:194] Registered new tserver with Master: e41e35b308f44384a382e29ad34bc838 (127.29.159.67:34061)
I20260927 18:47:43.226723 8090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:36575
I20260927 18:47:43.236763 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:47:43.246045 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.238633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.29.159.126:44021)
0927 18:47:43.238790 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:43.238795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:43.238943 (+ 148us) server_negotiation.cc:407] Connection header received
0927 18:47:43.239102 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:43.239104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:43.239143 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:43.239219 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:47:43.239933 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.240442 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.241251 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.241452 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.241525 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.242802 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:47:43.242805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:47:43.242807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:47:43.242827 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:47:43.244440 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:43.244839 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:43.244841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:43.244842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:43.244912 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:47:43.245222 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:47:43.245224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:47:43.245225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:47:43.245431 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:47:43.245531 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:43.245755 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 18:47:43.245843 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:47:43.248888 8090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:47:43.252193 8090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:47:43.266342 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.260652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:45623 (local address 127.0.0.1:42348)
0927 18:47:43.261428 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:43.261443 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:43.261537 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:43.262273 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:43.262279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:43.262312 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:43.262559 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.262570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.263696 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.263700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.264958 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.264967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.265783 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.265787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.265910 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.265947 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:47:43.266055 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 18:47:43.266182 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":258,"spinlock_wait_cycles":5120,"thread_start_us":106,"threads_started":1}
I20260927 18:47:43.266346 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.260875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.29.159.66:45623)
0927 18:47:43.261288 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:43.261303 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:43.261640 (+ 337us) server_negotiation.cc:407] Connection header received
0927 18:47:43.261728 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:43.261734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:43.261947 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:43.262095 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:43.262704 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.263553 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.265085 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.265656 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.265785 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.265943 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:43.266083 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 18:47:43.266182 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":116,"threads_started":1}
I20260927 18:47:43.266785 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.260652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:44213 (local address 127.0.0.1:52520)
0927 18:47:43.261267 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:43.261316 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:43.261458 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:43.262273 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:43.262279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:43.262310 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:43.262563 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.262572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.264121 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.264126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.265130 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.265139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.266369 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.266373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.266477 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.266499 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:47:43.266560 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 18:47:43.266631 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":66,"spinlock_wait_cycles":5120,"thread_start_us":80,"threads_started":1}
I20260927 18:47:43.267364 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.260771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.29.159.65:44213)
0927 18:47:43.261752 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:43.261770 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:43.261792 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:47:43.261868 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:43.261873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:43.262103 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:43.262376 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:43.262696 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.263910 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.265259 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.266083 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.266184 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.266310 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:43.267107 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 18:47:43.267209 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":101,"threads_started":1}
I20260927 18:47:43.268111 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.261390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:34061 (local address 127.0.0.1:58874)
0927 18:47:43.262070 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:43.262082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:43.262166 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:43.262583 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:43.262585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:43.262604 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:43.262703 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.262709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.265285 (+ 2576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.265287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.266378 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.266386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.267699 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.267702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.267821 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.267845 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:47:43.267926 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:47:43.267992 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":84,"threads_started":1}
I20260927 18:47:43.270143 8350 tablet_service.cc:1467] Processing CreateTablet for tablet 7587d7430642438392085df8c5d1fde5 (DEFAULT_TABLE table=test-table [id=f631bbb7e4374865ac4a1b265614cbbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:43.270144 8216 tablet_service.cc:1467] Processing CreateTablet for tablet 7587d7430642438392085df8c5d1fde5 (DEFAULT_TABLE table=test-table [id=f631bbb7e4374865ac4a1b265614cbbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:43.271100 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.261775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.29.159.67:34061)
0927 18:47:43.262167 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:43.262177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:43.262199 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:47:43.262277 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:43.262288 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:43.262459 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:43.262609 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:43.262790 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.263920 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.266508 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.267361 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.267504 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.267644 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:43.270862 (+ 3218us) server_negotiation.cc:299] Negotiation successful
0927 18:47:43.270970 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":89,"threads_started":1}
I20260927 18:47:43.271323 8216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7587d7430642438392085df8c5d1fde5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:43.271354 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7587d7430642438392085df8c5d1fde5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:43.273754 8484 tablet_service.cc:1467] Processing CreateTablet for tablet 7587d7430642438392085df8c5d1fde5 (DEFAULT_TABLE table=test-table [id=f631bbb7e4374865ac4a1b265614cbbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:47:43.274609 8484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7587d7430642438392085df8c5d1fde5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:47:43.278702 8567 tablet_bootstrap.cc:492] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b: Bootstrap starting.
I20260927 18:47:43.279892 8568 tablet_bootstrap.cc:492] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5: Bootstrap starting.
I20260927 18:47:43.281315 8567 tablet_bootstrap.cc:654] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:43.281520 8569 tablet_bootstrap.cc:492] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838: Bootstrap starting.
I20260927 18:47:43.282179 8567 log.cc:824] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:43.282300 8568 tablet_bootstrap.cc:654] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:43.283093 8568 log.cc:824] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:43.284058 8569 tablet_bootstrap.cc:654] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838: Neither blocks nor log segments found. Creating new log.
I20260927 18:47:43.284385 8567 tablet_bootstrap.cc:492] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b: No bootstrap required, opened a new log
I20260927 18:47:43.284569 8567 ts_tablet_manager.cc:1397] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:47:43.284637 8569 log.cc:824] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838: Log is configured to *not* fsync() on all Append() calls
I20260927 18:47:43.285000 8568 tablet_bootstrap.cc:492] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5: No bootstrap required, opened a new log
I20260927 18:47:43.285156 8568 ts_tablet_manager.cc:1397] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 18:47:43.286142 8569 tablet_bootstrap.cc:492] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838: No bootstrap required, opened a new log
I20260927 18:47:43.286332 8569 ts_tablet_manager.cc:1397] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:47:43.289039 8569 raft_consensus.cc:348] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0d2f6f392449ea44e3820399b204b" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 44213 } } peers { permanent_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45623 } } peers { permanent_uuid: "e41e35b308f44384a382e29ad34bc838" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34061 } }
I20260927 18:47:43.288970 8568 raft_consensus.cc:348] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0d2f6f392449ea44e3820399b204b" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 44213 } } peers { permanent_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45623 } } peers { permanent_uuid: "e41e35b308f44384a382e29ad34bc838" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34061 } }
I20260927 18:47:43.288970 8567 raft_consensus.cc:348] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0d2f6f392449ea44e3820399b204b" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 44213 } } peers { permanent_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45623 } } peers { permanent_uuid: "e41e35b308f44384a382e29ad34bc838" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34061 } }
I20260927 18:47:43.289258 8568 raft_consensus.cc:374] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:43.289258 8567 raft_consensus.cc:374] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:43.289258 8569 raft_consensus.cc:374] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:47:43.289317 8569 raft_consensus.cc:729] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41e35b308f44384a382e29ad34bc838, State: Initialized, Role: FOLLOWER
I20260927 18:47:43.289317 8567 raft_consensus.cc:729] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab0d2f6f392449ea44e3820399b204b, State: Initialized, Role: FOLLOWER
I20260927 18:47:43.289317 8568 raft_consensus.cc:729] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3cf1fe7c51477aaa86cdbb545134a5, State: Initialized, Role: FOLLOWER
I20260927 18:47:43.289770 8567 consensus_queue.cc:260] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0d2f6f392449ea44e3820399b204b" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 44213 } } peers { permanent_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45623 } } peers { permanent_uuid: "e41e35b308f44384a382e29ad34bc838" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34061 } }
I20260927 18:47:43.289786 8568 consensus_queue.cc:260] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0d2f6f392449ea44e3820399b204b" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 44213 } } peers { permanent_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45623 } } peers { permanent_uuid: "e41e35b308f44384a382e29ad34bc838" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34061 } }
I20260927 18:47:43.289784 8569 consensus_queue.cc:260] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0d2f6f392449ea44e3820399b204b" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 44213 } } peers { permanent_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45623 } } peers { permanent_uuid: "e41e35b308f44384a382e29ad34bc838" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34061 } }
I20260927 18:47:43.290729 8567 ts_tablet_manager.cc:1428] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260927 18:47:43.290729 8568 ts_tablet_manager.cc:1428] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 18:47:43.290771 8569 ts_tablet_manager.cc:1428] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 18:47:43.291081 8416 heartbeater.cc:503] Master 127.29.159.126:44021 was elected leader, sending a full tablet report...
I20260927 18:47:43.291081 8550 heartbeater.cc:503] Master 127.29.159.126:44021 was elected leader, sending a full tablet report...
I20260927 18:47:43.291095 8282 heartbeater.cc:503] Master 127.29.159.126:44021 was elected leader, sending a full tablet report...
I20260927 18:47:43.331154 8574 raft_consensus.cc:482] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:47:43.331400 8574 raft_consensus.cc:504] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0d2f6f392449ea44e3820399b204b" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 44213 } } peers { permanent_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45623 } } peers { permanent_uuid: "e41e35b308f44384a382e29ad34bc838" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34061 } }
I20260927 18:47:43.332623 8574 leader_election.cc:288] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ab0d2f6f392449ea44e3820399b204b (127.29.159.65:44213), e41e35b308f44384a382e29ad34bc838 (127.29.159.67:34061)
I20260927 18:47:43.336493 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.332899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:44213 (local address 127.29.159.66:48405)
0927 18:47:43.333043 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:43.333113 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:43.333263 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:43.333589 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:43.333592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:43.333611 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:43.333739 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.333745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.334481 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.334487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.335369 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.335378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.336053 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.336057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.336166 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.336188 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:47:43.336286 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:47:43.336359 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":30}
I20260927 18:47:43.336493 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.332980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:48405 (local address 127.29.159.65:44213)
0927 18:47:43.333112 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:43.333117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:43.333199 (+ 82us) server_negotiation.cc:407] Connection header received
0927 18:47:43.333400 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:43.333404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:43.333461 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:43.333576 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:43.333855 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.334369 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.335520 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.335931 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.336062 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.336117 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:43.336301 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 18:47:43.336370 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:47:43.336876 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.333359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:41929 (local address 127.29.159.67:34061)
0927 18:47:43.333498 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:43.333503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:43.333704 (+ 201us) server_negotiation.cc:407] Connection header received
0927 18:47:43.333829 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:43.333832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:43.333882 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:43.333966 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:47:43.334219 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.334705 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.335886 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.336283 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.336410 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.336458 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:43.336683 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 18:47:43.336763 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:47:43.337225 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.333215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:34061 (local address 127.29.159.66:41929)
0927 18:47:43.333618 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:43.333628 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:47:43.333740 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:47:43.333973 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:47:43.333976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:47:43.333991 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:47:43.334104 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.334118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.334810 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.334815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.335768 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:47:43.335776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.336400 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.336404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:47:43.336501 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.336522 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:47:43.337022 (+ 500us) client_negotiation.cc:240] Negotiation successful
0927 18:47:43.337094 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":20,"thread_start_us":153,"threads_started":1}
I20260927 18:47:43.337210 8236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7587d7430642438392085df8c5d1fde5" candidate_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab0d2f6f392449ea44e3820399b204b" is_pre_election: true
I20260927 18:47:43.337545 8236 raft_consensus.cc:2454] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc3cf1fe7c51477aaa86cdbb545134a5 in term 0.
I20260927 18:47:43.338078 8303 leader_election.cc:302] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ab0d2f6f392449ea44e3820399b204b, fc3cf1fe7c51477aaa86cdbb545134a5; no voters:
I20260927 18:47:43.338014 8504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7587d7430642438392085df8c5d1fde5" candidate_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41e35b308f44384a382e29ad34bc838" is_pre_election: true
I20260927 18:47:43.338364 8504 raft_consensus.cc:2454] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc3cf1fe7c51477aaa86cdbb545134a5 in term 0.
I20260927 18:47:43.338373 8574 raft_consensus.cc:2781] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:47:43.338469 8574 raft_consensus.cc:482] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:47:43.338547 8574 raft_consensus.cc:3037] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:43.339798 8574 raft_consensus.cc:504] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0d2f6f392449ea44e3820399b204b" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 44213 } } peers { permanent_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45623 } } peers { permanent_uuid: "e41e35b308f44384a382e29ad34bc838" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34061 } }
I20260927 18:47:43.340142 8574 leader_election.cc:288] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [CANDIDATE]: Term 1 election: Requested vote from peers 6ab0d2f6f392449ea44e3820399b204b (127.29.159.65:44213), e41e35b308f44384a382e29ad34bc838 (127.29.159.67:34061)
I20260927 18:47:43.340479 8236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7587d7430642438392085df8c5d1fde5" candidate_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab0d2f6f392449ea44e3820399b204b"
I20260927 18:47:43.340543 8504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7587d7430642438392085df8c5d1fde5" candidate_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41e35b308f44384a382e29ad34bc838"
I20260927 18:47:43.340610 8236 raft_consensus.cc:3037] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:43.340688 8504 raft_consensus.cc:3037] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:47:43.341928 8504 raft_consensus.cc:2454] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc3cf1fe7c51477aaa86cdbb545134a5 in term 1.
I20260927 18:47:43.341928 8236 raft_consensus.cc:2454] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc3cf1fe7c51477aaa86cdbb545134a5 in term 1.
I20260927 18:47:43.342235 8303 leader_election.cc:302] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e41e35b308f44384a382e29ad34bc838, fc3cf1fe7c51477aaa86cdbb545134a5; no voters:
I20260927 18:47:43.342396 8574 raft_consensus.cc:2781] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:47:43.342650 8574 raft_consensus.cc:686] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 1 LEADER]: Becoming Leader. State: Replica: fc3cf1fe7c51477aaa86cdbb545134a5, State: Running, Role: LEADER
I20260927 18:47:43.342988 8574 consensus_queue.cc:237] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0d2f6f392449ea44e3820399b204b" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 44213 } } peers { permanent_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45623 } } peers { permanent_uuid: "e41e35b308f44384a382e29ad34bc838" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34061 } }
I20260927 18:47:43.346434 8089 catalog_manager.cc:5495] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc3cf1fe7c51477aaa86cdbb545134a5 (127.29.159.66). New cstate: current_term: 1 leader_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ab0d2f6f392449ea44e3820399b204b" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 44213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc3cf1fe7c51477aaa86cdbb545134a5" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41e35b308f44384a382e29ad34bc838" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34061 } health_report { overall_health: UNKNOWN } } }
I20260927 18:47:43.365598 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.361836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.29.159.66:45623)
0927 18:47:43.361983 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:47:43.361987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:47:43.362002 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:47:43.362167 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:47:43.362170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:47:43.362235 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:47:43.362352 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:47:43.362644 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.363154 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.364111 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:47:43.364343 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:47:43.364455 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:47:43.364773 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:47:43.365204 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:47:43.365343 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:47:43.365379 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 18:47:43.365434 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:47:43.371659 8504 raft_consensus.cc:1261] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838 [term 1 FOLLOWER]: Refusing update from remote peer fc3cf1fe7c51477aaa86cdbb545134a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:43.371659 8236 raft_consensus.cc:1261] T 7587d7430642438392085df8c5d1fde5 P 6ab0d2f6f392449ea44e3820399b204b [term 1 FOLLOWER]: Refusing update from remote peer fc3cf1fe7c51477aaa86cdbb545134a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:47:43.372438 8574 consensus_queue.cc:1035] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab0d2f6f392449ea44e3820399b204b" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 44213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:43.372692 8577 consensus_queue.cc:1035] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41e35b308f44384a382e29ad34bc838" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 34061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:47:43.379220 8580 mvcc.cc:205] Tried to move back new op lower bound from 7334030800362115072 to 7334030800255848448. Current Snapshot: MvccSnapshot[applied={T|T < 7334030800362115072}]
I20260927 18:47:43.381168 8582 mvcc.cc:205] Tried to move back new op lower bound from 7334030800362115072 to 7334030800255848448. Current Snapshot: MvccSnapshot[applied={T|T < 7334030800362115072}]
I20260927 18:47:43.381531 8581 mvcc.cc:205] Tried to move back new op lower bound from 7334030800362115072 to 7334030800255848448. Current Snapshot: MvccSnapshot[applied={T|T < 7334030800362115072}]
I20260927 18:47:43.389282 8089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:43.389436 8089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:47:43.391957 8089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0580fdd11724e03a76a2f4ce4bf0a76: Sending DeleteTablet for 3 replicas of tablet 7587d7430642438392085df8c5d1fde5
I20260927 18:47:43.392845 8350 tablet_service.cc:1514] Processing DeleteTablet for tablet 7587d7430642438392085df8c5d1fde5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:43 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:42348
I20260927 18:47:43.392843 8216 tablet_service.cc:1514] Processing DeleteTablet for tablet 7587d7430642438392085df8c5d1fde5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:43 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:52520
I20260927 18:47:43.393404 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 8153
I20260927 18:47:43.393540 8484 tablet_service.cc:1514] Processing DeleteTablet for tablet 7587d7430642438392085df8c5d1fde5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:47:43 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:58874
I20260927 18:47:43.394598 8592 tablet_replica.cc:330] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838: stopping tablet replica
I20260927 18:47:43.394942 8592 raft_consensus.cc:2229] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:47:43.395251 8592 raft_consensus.cc:2258] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:43.395772 8591 tablet_replica.cc:330] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5: stopping tablet replica
I20260927 18:47:43.396046 8591 raft_consensus.cc:2229] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:47:43.396401 8591 raft_consensus.cc:2258] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:47:43.397483 8592 ts_tablet_manager.cc:1905] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:43.397656 8591 ts_tablet_manager.cc:1905] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:47:43.400324 8591 ts_tablet_manager.cc:1918] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:43.400360 8592 ts_tablet_manager.cc:1918] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:47:43.400422 8591 log.cc:1196] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/7587d7430642438392085df8c5d1fde5
I20260927 18:47:43.400449 8592 log.cc:1196] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/7587d7430642438392085df8c5d1fde5
I20260927 18:47:43.400755 8591 ts_tablet_manager.cc:1939] T 7587d7430642438392085df8c5d1fde5 P fc3cf1fe7c51477aaa86cdbb545134a5: Deleting consensus metadata
I20260927 18:47:43.400806 8592 ts_tablet_manager.cc:1939] T 7587d7430642438392085df8c5d1fde5 P e41e35b308f44384a382e29ad34bc838: Deleting consensus metadata
I20260927 18:47:43.401914 8076 catalog_manager.cc:4841] TS fc3cf1fe7c51477aaa86cdbb545134a5 (127.29.159.66:45623): tablet 7587d7430642438392085df8c5d1fde5 (table test-table [id=f631bbb7e4374865ac4a1b265614cbbe]) successfully deleted
I20260927 18:47:43.401988 8075 catalog_manager.cc:4841] TS e41e35b308f44384a382e29ad34bc838 (127.29.159.67:34061): tablet 7587d7430642438392085df8c5d1fde5 (table test-table [id=f631bbb7e4374865ac4a1b265614cbbe]) successfully deleted
W20260927 18:47:43.404490 8075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:44213 (error 108)
I20260927 18:47:43.405241 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 8287
I20260927 18:47:43.405522 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:47:43.405234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:44213 (local address 127.0.0.1:52524)
0927 18:47:43.405360 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 18:47:43.405430 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:44213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:47:43.407151 8075 catalog_manager.cc:4584] TS 6ab0d2f6f392449ea44e3820399b204b (127.29.159.65:44213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7587d7430642438392085df8c5d1fde5: Network error: Client connection negotiation failed: client connection to 127.29.159.65:44213: connect: Connection refused (error 111)
I20260927 18:47:43.410100 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 8421
I20260927 18:47:43.415371 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 8059
2026-09-27T18:47:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2342 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:47:43 dist-test-slave-8tg2 krb5kdc[8596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:47:43 dist-test-slave-8tg2 krb5kdc[8596](info): set up 2 sockets
Sep 27 18:47:43 dist-test-slave-8tg2 krb5kdc[8596](info): commencing operation
krb5kdc: starting...
W20260927 18:47:44.460594 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:47:44 dist-test-slave-8tg2 krb5kdc[8596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:47:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:47:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.159.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.159.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/rangeradmin_127.29.159.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/rangeradmin_127.29.159.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.159.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.159.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/rangerlookup_127.29.159.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/rangerlookup_127.29.159.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.159.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.159.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/HTTP_127.29.159.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/HTTP_127.29.159.84@KRBTEST.COM.keytab.
I20260927 18:47:44.639487 30333 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskclkeqe/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/postgres -l logfile start
2026-09-27 18:47:47.155 UTC [8626] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:47:47.155 UTC [8626] LOG: listening on IPv4 address "127.29.159.84", port 39467
2026-09-27 18:47:47.158 UTC [8626] LOG: listening on Unix socket "/tmp/.s.PGSQL.39467"
2026-09-27 18:47:47.193 UTC [8629] LOG: database system was shut down at 2026-09-27 18:47:45 UTC
2026-09-27 18:47:47.197 UTC [8626] LOG: database system is ready to accept connections
I20260927 18:47:48.140295 30333 mini_postgres.cc:96] Postgres bound to 39467
2026-09-27 18:47:48.146 UTC [8637] FATAL: database "slave" does not exist
127.29.159.84:39467 - accepting connections
I20260927 18:47:48.147066 30333 mini_ranger.cc:161] Starting Ranger...
I20260927 18:47:48.161664 30333 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:47:48.611706 30333 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:47:48.611819 30333 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-admin
2026-09-27 18:47:48,886 [I] DB FLAVOR :POSTGRES
2026-09-27 18:47:48,886 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:47:48,886 [I] Checking connection..
2026-09-27 18:47:48,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:47:49,123 [I] Checking connection passed.
2026-09-27 18:47:49,124 [I] --------- Verifying version history table ---------
2026-09-27 18:47:49,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:47:49,373 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:47:49,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:47:49,633 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:47:49,633 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:47:49,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:47:49.857 UTC [8701] WARNING: there is no transaction in progress
2026-09-27 18:47:49,863 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:47:49,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:47:50,124 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:47:50,124 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:47:50,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:47:50,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:47:50,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8tg2.c.gcp-upstream.internal','N') ;"
2026-09-27 18:47:50,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:47:51,140 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:47:51,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:47:51,378 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:47:51,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:47:51,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:47:51,869 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:47:51,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:47:52.812 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:52.829 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:52.845 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:52.860 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:52.876 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:52.896 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:52.956 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:52.963 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:52.979 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:52.989 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:53.000 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:53.015 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:53.024 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:53.031 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:53.038 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:53.045 UTC [8809] WARNING: there is no transaction in progress
2026-09-27 18:47:53,709 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:47:53,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:47:53,939 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:47:53,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:47:54,198 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:47:54,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:47:54,454 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:47:54,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:47:54,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:47:54,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8tg2.c.gcp-upstream.internal';"
2026-09-27 18:47:55,209 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:47:55,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:47:55,456 [I] DB_PATCHES have already been applied
I20260927 18:47:55.462791 30333 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-admin:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskclkeqe/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:47:55.462873 30333 mini_ranger.cc:193] Using host: 127.29.159.84
I20260927 18:47:55.466673 30333 mini_ranger.cc:227] Ranger admin URL: http://127.29.159.84:36363
Sep 27, 2026 6:47:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:47:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:47:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:47:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:47:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:47:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:47:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:47:56.561563 30333 mini_ranger.cc:160] Time spent starting Ranger: real 8.415s user 0.003s sys 0.007s
Sep 27, 2026 6:48:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:48:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 6:48:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:48:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:48:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:48:17.227308 30333 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.159.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.159.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/rangerkms_127.29.159.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/rangerkms_127.29.159.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.159.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.159.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.159.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/HTTP_127.29.159.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.159.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/HTTP_127.29.159.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:48:17 dist-test-slave-8tg2 krb5kdc[8596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534897, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:48:17.310899 30333 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:48:17.320025 30333 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:48:18.171077 30333 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:48:18.171175 30333 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-kms
I20260927 18:48:18.171293 30333 mini_ranger_kms.cc:227] Using postgres at 127.29.159.84:39467
2026-09-27 18:48:18,341 [I] DB FLAVOR :POSTGRES
2026-09-27 18:48:18,341 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:48:18,342 [I] Checking connection
2026-09-27 18:48:18,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:48:18,602 [I] connection success
2026-09-27 18:48:18,602 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:48:18,602 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:48:18,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:48:18,864 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:48:18,865 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:48:18,865 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:48:18,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:39467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:48:19,145 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:48:19.587277 30333 mini_ranger_kms.cc:310] Created kms service
I20260927 18:48:19.754608 30333 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:48:19.795974 30333 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:48:19 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534897, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:48:20.165853 30333 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:48:20.166013 30333 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-kms:/tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:48:20.166033 30333 mini_ranger_kms.cc:240] Using host: 127.29.159.84
I20260927 18:48:20.170260 30333 mini_ranger_kms.cc:276] Ranger KMS PID: 8989
I20260927 18:48:20.170404 30333 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.159.84:41509
Sep 27, 2026 6:48:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:48:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:48:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:48:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:48:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:48:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 18:48:22.399621 30333 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.089s user 0.012s sys 0.143s
I20260927 18:48:22.399756 30333 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:48:22.399886 30333 mini_ranger_kms.cc:401] 127.29.159.84:41509/kms/v1/keys
Sep 27 18:48:22 dist-test-slave-8tg2 krb5kdc[8596](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534902, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.159.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 18:48:23 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534902, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.159.84@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
Sep 27 18:48:23 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534902, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.159.84@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
Sep 27 18:48:25 dist-test-slave-8tg2 krb5kdc[8596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:48:25.547248 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41861
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:41861
--ranger_config_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:41509/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:25.676798 9040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:25.677089 9040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:25.677162 9040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:25.681299 9040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:48:25.681386 9040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:25.681422 9040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:25.681458 9040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:48:25.681491 9040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:48:25.687175 9040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.159.84:41509/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:41861
--ranger_config_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:48:25.688380 9040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:25.689493 9040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:25.698832 9048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:25.698861 9045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:25.699013 9040 server_base.cc:1034] running on GCE node
W20260927 18:48:25.699023 9046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:25.699939 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:25.701448 9040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:25.702652 9040 hybrid_clock.cc:648] HybridClock initialized: now 1790534905702625 us; error 47 us; skew 500 ppm
Sep 27 18:48:25 dist-test-slave-8tg2 krb5kdc[8596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534905, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:25.705957 9040 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:48:25.707084 9040 webserver.cc:460] Webserver started at http://127.29.159.126:40077/ using document root <none> and password file <none>
I20260927 18:48:25.707813 9040 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:25.707896 9040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:25.708161 9040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:48:25 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534905, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:48:25.768996 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "d1f91c425cdc47b3b539bc8a537b097c"
format_stamp: "Formatted at 2026-09-27 18:48:25 on dist-test-slave-8tg2"
server_key: "e23957e6560943d6d5f301179e5df655"
server_key_iv: "fcfa6de0c9f33fd2cf3a7cd621743cef"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:25.769619 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "d1f91c425cdc47b3b539bc8a537b097c"
format_stamp: "Formatted at 2026-09-27 18:48:25 on dist-test-slave-8tg2"
server_key: "e23957e6560943d6d5f301179e5df655"
server_key_iv: "fcfa6de0c9f33fd2cf3a7cd621743cef"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:25.773919 9040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:48:25.791299 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:25.792802 9040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:48:25.792936 9040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "d1f91c425cdc47b3b539bc8a537b097c"
format_stamp: "Formatted at 2026-09-27 18:48:25 on dist-test-slave-8tg2"
server_key: "e23957e6560943d6d5f301179e5df655"
server_key_iv: "fcfa6de0c9f33fd2cf3a7cd621743cef"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:25.793074 9040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:25.816483 9040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:25.819650 9040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:25.819861 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:25.827694 9040 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:41861
I20260927 18:48:25.827724 9107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:41861 every 8 connection(s)
I20260927 18:48:25.828644 9040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:48:25.831980 9108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:25.836063 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 9040
I20260927 18:48:25.836215 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:48:25.836501 30333 external_mini_cluster.cc:1428] Setting key c8137dcc7c2369fcffd92b3db477dc7f
I20260927 18:48:25.838889 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c: Bootstrap starting.
I20260927 18:48:25.841547 9108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:25.842365 9108 log.cc:824] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:25.844482 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c: No bootstrap required, opened a new log
Sep 27 18:48:25 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:48:25.849079 9108 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f91c425cdc47b3b539bc8a537b097c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41861 } }
I20260927 18:48:25.849407 9108 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:25.849521 9108 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f91c425cdc47b3b539bc8a537b097c, State: Initialized, Role: FOLLOWER
I20260927 18:48:25.850075 9108 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f91c425cdc47b3b539bc8a537b097c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41861 } }
I20260927 18:48:25.850271 9108 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:48:25.850378 9108 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:48:25.850499 9108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:25.851363 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:25.837947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.29.159.126:41861)
0927 18:48:25.838366 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:25.838382 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:25.838418 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:48:25.839071 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:25.839089 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:25.839415 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:25.839793 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:25.840583 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:25.841551 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:25.842479 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:25.843132 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:25.843305 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:25.846346 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:25.846353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:25.846366 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:25.846390 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:25.848085 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:25.848536 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:25.848546 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:25.848551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:25.848651 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:25.848955 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:25.848960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:25.848961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:25.849339 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:25.850181 (+ 842us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:25.850368 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 18:48:25.850660 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":143,"threads_started":1}
I20260927 18:48:25.851713 9108 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f91c425cdc47b3b539bc8a537b097c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41861 } }
I20260927 18:48:25.852073 9108 leader_election.cc:302] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1f91c425cdc47b3b539bc8a537b097c; no voters:
I20260927 18:48:25.852407 9108 leader_election.cc:288] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:48:25.852847 9114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:48:25.853098 9114 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [term 1 LEADER]: Becoming Leader. State: Replica: d1f91c425cdc47b3b539bc8a537b097c, State: Running, Role: LEADER
I20260927 18:48:25.853439 9114 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f91c425cdc47b3b539bc8a537b097c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41861 } }
I20260927 18:48:25.853792 9108 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:48:25.854923 9114 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1f91c425cdc47b3b539bc8a537b097c. Latest consensus state: current_term: 1 leader_uuid: "d1f91c425cdc47b3b539bc8a537b097c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f91c425cdc47b3b539bc8a537b097c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41861 } } }
I20260927 18:48:25.855126 9114 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:25.855352 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1f91c425cdc47b3b539bc8a537b097c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f91c425cdc47b3b539bc8a537b097c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41861 } } }
I20260927 18:48:25.855458 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c [sys.catalog]: This master's current role is: LEADER
I20260927 18:48:25.855624 9122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:48:25.859050 9122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:48:25.859495 9108 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:48:25.867537 9122 catalog_manager.cc:1347] Generated new cluster ID: 184c66e034ec4a99a095c076be425a0d
I20260927 18:48:25.867739 9122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:48:25.899199 9122 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:48:25.900681 9122 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:48:25.906380 9122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c: Generated new TSK 0
I20260927 18:48:25.907382 9122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:48:26.676983 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:41509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41861
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:26.834462 9153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:26.834803 9153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:26.834869 9153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:26.840832 9153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:26.841001 9153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:26.841156 9153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:48:26.848381 9153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.159.84:41509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:48:26.849956 9153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:26.851348 9153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:26.864099 9161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:26.864537 9160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:26.864718 9163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:26.865581 9153 server_base.cc:1034] running on GCE node
I20260927 18:48:26.866137 9153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:26.866941 9153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:26.868131 9153 hybrid_clock.cc:648] HybridClock initialized: now 1790534906868061 us; error 85 us; skew 500 ppm
Sep 27 18:48:26 dist-test-slave-8tg2 krb5kdc[8596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534906, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:26.871882 9153 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:48:26.873041 9153 webserver.cc:460] Webserver started at http://127.29.159.65:37029/ using document root <none> and password file <none>
I20260927 18:48:26.873775 9153 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:26.873886 9153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:26.874126 9153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:48:26 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534906, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:48:26.893172 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "d0bb554aa93e46bebf38470509dadc1e"
format_stamp: "Formatted at 2026-09-27 18:48:26 on dist-test-slave-8tg2"
server_key: "3eeaea831da3b4f300f3e56dfaa1fdc6"
server_key_iv: "42127ea567d73c011b0802ba79552a5f"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:26.893842 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "d0bb554aa93e46bebf38470509dadc1e"
format_stamp: "Formatted at 2026-09-27 18:48:26 on dist-test-slave-8tg2"
server_key: "3eeaea831da3b4f300f3e56dfaa1fdc6"
server_key_iv: "42127ea567d73c011b0802ba79552a5f"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:26.900198 9153 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:48:26.935925 9170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:26.938980 9153 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.003s sys 0.000s
I20260927 18:48:26.939137 9153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "d0bb554aa93e46bebf38470509dadc1e"
format_stamp: "Formatted at 2026-09-27 18:48:26 on dist-test-slave-8tg2"
server_key: "3eeaea831da3b4f300f3e56dfaa1fdc6"
server_key_iv: "42127ea567d73c011b0802ba79552a5f"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:26.939289 9153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:26.965590 9153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:26.969389 9153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:26.969643 9153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:26.970360 9153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:26.971475 9153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:26.971580 9153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:26.971695 9153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:26.971769 9153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:26.984489 9153 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:39105
I20260927 18:48:26.985695 9153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:48:26.986613 9285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:39105 every 8 connection(s)
I20260927 18:48:26.988554 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 9153
I20260927 18:48:26.988689 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:48:26.989027 30333 external_mini_cluster.cc:1428] Setting key 14c0c0a937899ed92ad9cf47d08bd7ec
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
Sep 27 18:48:26 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534906, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
I20260927 18:48:27.008906 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:26.990409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:42457 (local address 127.29.159.126:41861)
0927 18:48:26.991363 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:26.991368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:26.993261 (+ 1893us) server_negotiation.cc:407] Connection header received
0927 18:48:26.993315 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:26.993318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:26.993367 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:26.993447 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:26.994897 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:26.995801 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:26.997840 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:26.998205 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:26.998317 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.001467 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:27.001472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:27.001474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:27.001503 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:27.003535 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:27.004342 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:27.004345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:27.004347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:27.004434 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:27.004930 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:27.004933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:27.004935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:27.005225 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:27.005318 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:27.008582 (+ 3264us) server_negotiation.cc:299] Negotiation successful
0927 18:48:27.008709 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":58,"threads_started":1}
I20260927 18:48:27.010330 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:26.989292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41861 (local address 127.29.159.65:42457)
0927 18:48:26.991822 (+ 2530us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:26.991878 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:26.992903 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:26.993745 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:26.993754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:26.994090 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:26.994735 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:26.994752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:26.996229 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:26.996232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:26.997702 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:26.997713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:26.998535 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:26.998539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:26.998733 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:26.999408 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:26.999455 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:27.001256 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:27.003834 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:27.003841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:27.003858 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:27.004217 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:27.004601 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:27.004604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:27.004606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:27.004810 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:27.007311 (+ 2501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:27.007319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:27.007677 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 18:48:27.007966 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 18:48:27.008338 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2302,"thread_start_us":131,"threads_started":1}
I20260927 18:48:27.011998 9286 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41861
I20260927 18:48:27.012238 9286 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:27.012797 9286 heartbeater.cc:511] Master 127.29.159.126:41861 requested a full tablet report, sending...
I20260927 18:48:27.014496 9072 ts_manager.cc:194] Registered new tserver with Master: d0bb554aa93e46bebf38470509dadc1e (127.29.159.65:39105)
I20260927 18:48:27.016007 9072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:42457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:48:27.036636 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:41509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41861
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:48:27 dist-test-slave-8tg2 krb5kdc[8596](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534907, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:48:27.185093 9292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:27.185491 9292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:27.185562 9292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:27.191726 9292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:27.191817 9292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:27.191907 9292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:48:27.200022 9292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.159.84:41509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:48:27.201532 9292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:27.202953 9292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:48:27.215652 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:27.215873 9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:27.215922 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:27.217368 9292 server_base.cc:1034] running on GCE node
I20260927 18:48:27.217906 9292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:27.218802 9292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:27.220003 9292 hybrid_clock.cc:648] HybridClock initialized: now 1790534907219977 us; error 41 us; skew 500 ppm
Sep 27 18:48:27 dist-test-slave-8tg2 krb5kdc[8596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534907, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:27.224186 9292 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:48:27.225344 9292 webserver.cc:460] Webserver started at http://127.29.159.66:44013/ using document root <none> and password file <none>
I20260927 18:48:27.226136 9292 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:27.226287 9292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:27.226634 9292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:48:27 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534907, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:48:27.242832 9292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "e4f7b1248d864c9b9620d5b2f506e06c"
format_stamp: "Formatted at 2026-09-27 18:48:27 on dist-test-slave-8tg2"
server_key: "d5c960e7eb8cac13be6e55652e337c77"
server_key_iv: "37b184c89922bd6fbe214e37d8f84272"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:27.243644 9292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "e4f7b1248d864c9b9620d5b2f506e06c"
format_stamp: "Formatted at 2026-09-27 18:48:27 on dist-test-slave-8tg2"
server_key: "d5c960e7eb8cac13be6e55652e337c77"
server_key_iv: "37b184c89922bd6fbe214e37d8f84272"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:27.249353 9292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:48:27.270538 9308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:27.278592 9292 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.000s
I20260927 18:48:27.278858 9292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "e4f7b1248d864c9b9620d5b2f506e06c"
format_stamp: "Formatted at 2026-09-27 18:48:27 on dist-test-slave-8tg2"
server_key: "d5c960e7eb8cac13be6e55652e337c77"
server_key_iv: "37b184c89922bd6fbe214e37d8f84272"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:27.279122 9292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:27.294430 9292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:27.298094 9292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:27.298396 9292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:27.299198 9292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:27.300247 9292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:27.300326 9292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:27.300424 9292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:27.300467 9292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:27.313287 9292 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:34755
I20260927 18:48:27.314579 9292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:48:27.314886 9421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:34755 every 8 connection(s)
I20260927 18:48:27.317269 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 9292
I20260927 18:48:27.317404 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:48:27.317751 30333 external_mini_cluster.cc:1428] Setting key ffe34acdc1a6863994447f4f0419565d
Sep 27 18:48:27 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534907, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:48:27.331568 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.316851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:59477 (local address 127.29.159.126:41861)
0927 18:48:27.317003 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:27.317007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:27.318864 (+ 1857us) server_negotiation.cc:407] Connection header received
0927 18:48:27.319972 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:27.319975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:27.320031 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:27.320103 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:27.321275 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.322143 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.323379 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.323828 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.323946 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.327105 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:27.327108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:27.327110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:27.327134 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:27.329038 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:27.329670 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:27.329675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:27.329677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:27.329774 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:27.330166 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:27.330172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:27.330177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:27.330427 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:27.330541 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:27.331139 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 18:48:27.331278 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:48:27.332528 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.317380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41861 (local address 127.29.159.66:59477)
0927 18:48:27.318675 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:27.318733 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:27.319736 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:27.320287 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:27.320293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:27.320562 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:27.321088 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.321110 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.322320 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.322325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.323251 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.323260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.324128 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.324135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.324381 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.325172 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:27.325201 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:27.326939 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:27.329175 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:27.329183 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:27.329200 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:27.329552 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:27.329879 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:27.329882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:27.329883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:27.330059 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:27.330541 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:27.330547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:27.330852 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 18:48:27.331062 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 18:48:27.331471 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"thread_start_us":167,"threads_started":1}
I20260927 18:48:27.333839 9422 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41861
I20260927 18:48:27.334096 9422 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:48:27.334687 9422 heartbeater.cc:511] Master 127.29.159.126:41861 requested a full tablet report, sending...
I20260927 18:48:27.335933 9072 ts_manager.cc:194] Registered new tserver with Master: e4f7b1248d864c9b9620d5b2f506e06c (127.29.159.66:34755)
I20260927 18:48:27.336660 9072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:59477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:48:27.359249 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:41509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41861
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:46711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:48:27.492702 9427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:48:27.493136 9427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:48:27.493259 9427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:48:27.499614 9427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:48:27.499749 9427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:48:27.499904 9427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:48:27.508118 9427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:46711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.159.84:41509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:48:27.510046 9427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:48:27.511533 9427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 18:48:27 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534907, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
W20260927 18:48:27.524392 9433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:27.524947 9435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:48:27.525211 9432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:48:27.526257 9427 server_base.cc:1034] running on GCE node
I20260927 18:48:27.526827 9427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:48:27.530721 9427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:48:27.533996 9427 hybrid_clock.cc:648] HybridClock initialized: now 1790534907533922 us; error 85 us; skew 500 ppm
Sep 27 18:48:27 dist-test-slave-8tg2 krb5kdc[8596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534907, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:48:27.537657 9427 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:48:27.538686 9427 webserver.cc:460] Webserver started at http://127.29.159.67:46741/ using document root <none> and password file <none>
I20260927 18:48:27.539539 9427 fs_manager.cc:359] Metadata directory not provided
I20260927 18:48:27.539644 9427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:48:27.539917 9427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:48:27 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534907, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:48:27.555989 9427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "09214af816b847c1a0bc3beb991c2794"
format_stamp: "Formatted at 2026-09-27 18:48:27 on dist-test-slave-8tg2"
server_key: "0ca7f6af875a55fb01427d2701fd8acd"
server_key_iv: "d407376435de6b9c9caf61c08708c66a"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:27.556798 9427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "09214af816b847c1a0bc3beb991c2794"
format_stamp: "Formatted at 2026-09-27 18:48:27 on dist-test-slave-8tg2"
server_key: "0ca7f6af875a55fb01427d2701fd8acd"
server_key_iv: "d407376435de6b9c9caf61c08708c66a"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:27.569005 9427 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260927 18:48:27.593930 9442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:27.595701 9427 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:48:27.595916 9427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "09214af816b847c1a0bc3beb991c2794"
format_stamp: "Formatted at 2026-09-27 18:48:27 on dist-test-slave-8tg2"
server_key: "0ca7f6af875a55fb01427d2701fd8acd"
server_key_iv: "d407376435de6b9c9caf61c08708c66a"
server_key_version: "kuduclusterkey@0"
I20260927 18:48:27.596153 9427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:48:27.618211 9427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:48:27.621448 9427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:48:27.621655 9427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:48:27.622259 9427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:48:27.623448 9427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:48:27.623539 9427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:27.623622 9427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:48:27.623662 9427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:48:27.633266 9427 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:32875
I20260927 18:48:27.633335 9555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:32875 every 8 connection(s)
I20260927 18:48:27.634384 9427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:48:27.637864 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 9427
I20260927 18:48:27.638000 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:48:27.638305 30333 external_mini_cluster.cc:1428] Setting key 268ddc85ad707fd12b68570d2bd7a0e7
Sep 27 18:48:27 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534907, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:48:27.649390 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.636446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:35617 (local address 127.29.159.126:41861)
0927 18:48:27.636573 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:27.636577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:27.637706 (+ 1129us) server_negotiation.cc:407] Connection header received
0927 18:48:27.638960 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:27.638964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:27.639019 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:27.639071 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:27.640511 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.641279 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.642401 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.642726 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.642836 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.645519 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:27.645523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:27.645527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:27.645558 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:27.647193 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:27.647727 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:27.647730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:27.647732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:27.647831 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:27.648243 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:27.648246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:27.648248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:27.648485 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:27.648590 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:27.649088 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 18:48:27.649216 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:48:27.650209 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.636961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41861 (local address 127.29.159.67:35617)
0927 18:48:27.637540 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:27.637596 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:27.638706 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:27.639534 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:27.639544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:27.639848 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:48:27.640342 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.640356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.641408 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.641411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.642270 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.642279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.642878 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.642881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.643023 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.643710 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:48:27.643732 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:48:27.645362 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:48:27.647294 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:27.647299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:27.647317 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:27.647631 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:27.647956 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:48:27.647959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:48:27.647961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:48:27.648134 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:48:27.648585 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:48:27.648591 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:48:27.648832 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 18:48:27.648987 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 18:48:27.649234 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":136,"threads_started":1}
I20260927 18:48:27.651564 9556 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41861
I20260927 18:48:27.651813 9556 heartbeater.cc:464] Registering TS with master...
I20260927 18:48:27.652289 9556 heartbeater.cc:511] Master 127.29.159.126:41861 requested a full tablet report, sending...
I20260927 18:48:27.653263 9072 ts_manager.cc:194] Registered new tserver with Master: 09214af816b847c1a0bc3beb991c2794 (127.29.159.67:32875)
I20260927 18:48:27.653891 9072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:35617
I20260927 18:48:27.664402 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:48:27.676231 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.666933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.29.159.126:41861)
0927 18:48:27.667083 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:27.667088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:27.667301 (+ 213us) server_negotiation.cc:407] Connection header received
0927 18:48:27.667493 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:27.667495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:27.667539 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:27.667605 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:48:27.668525 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.669169 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.670355 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.670700 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.670832 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.672569 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:48:27.672574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:48:27.672577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:48:27.672603 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:48:27.674357 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:27.674846 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:27.674851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:27.674853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:27.674939 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:48:27.675226 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:48:27.675229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:48:27.675231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:48:27.675484 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:48:27.675620 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:27.675871 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 18:48:27.675985 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:48:27.679484 9072 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:48:27.684034 9072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:48:27.703644 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.694666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:34755 (local address 127.0.0.1:33802)
0927 18:48:27.696547 (+ 1881us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:27.696601 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:27.696729 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:27.698642 (+ 1913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:27.698645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:27.698663 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:27.698756 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.698762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.700322 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.700326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.701841 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.701854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.703095 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.703099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.703211 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.703255 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 18:48:27.703352 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:48:27.703490 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1686,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260927 18:48:27.704520 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.694709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:39105 (local address 127.0.0.1:43710)
0927 18:48:27.696948 (+ 2239us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:27.696960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:27.697032 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:27.697854 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:27.697860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:27.697884 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:27.698024 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.698034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.700418 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.700421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.701890 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.701901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.704118 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.704122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.704212 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.704242 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:48:27.704317 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:48:27.704387 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2178,"mutex_wait_us":70,"thread_start_us":115,"threads_started":1}
I20260927 18:48:27.704901 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.695581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.29.159.66:34755)
0927 18:48:27.698178 (+ 2597us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:27.698189 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:27.698211 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:48:27.698280 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:27.698285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:27.698487 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:27.698898 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:27.699024 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.700177 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.702130 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.702959 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.704268 (+ 1309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.704429 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:27.704484 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 18:48:27.704762 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2475,"thread_start_us":123,"threads_started":1}
I20260927 18:48:27.705053 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.695565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.29.159.65:39105)
0927 18:48:27.697265 (+ 1700us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:27.697273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:27.697296 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:48:27.697468 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:27.697474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:27.697700 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:27.697868 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:27.698196 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.700275 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.702906 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.703769 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.703861 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.704014 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:27.704813 (+ 799us) server_negotiation.cc:299] Negotiation successful
0927 18:48:27.704917 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1588,"thread_start_us":97,"threads_started":1}
I20260927 18:48:27.708195 9356 tablet_service.cc:1467] Processing CreateTablet for tablet 6c206887f6c04aad83eb7d080724a36c (DEFAULT_TABLE table=test-table [id=71e7073c97b94d51a51a9ef61d55f952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:27.709466 9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c206887f6c04aad83eb7d080724a36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:27.713637 9220 tablet_service.cc:1467] Processing CreateTablet for tablet 6c206887f6c04aad83eb7d080724a36c (DEFAULT_TABLE table=test-table [id=71e7073c97b94d51a51a9ef61d55f952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:27.714740 9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c206887f6c04aad83eb7d080724a36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:27.720063 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.696339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:32875 (local address 127.0.0.1:59516)
0927 18:48:27.701832 (+ 5493us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:27.701846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:27.701945 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:27.715492 (+ 13547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:27.715496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:27.715518 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:27.715627 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.715633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.717084 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.717088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.718272 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.718283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.719585 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.719588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.719705 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.719732 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:48:27.719825 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:48:27.719908 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5424,"mutex_wait_us":500,"thread_start_us":82,"threads_started":1}
I20260927 18:48:27.720585 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.696564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.29.159.67:32875)
0927 18:48:27.701688 (+ 5124us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:27.701699 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:27.702036 (+ 337us) server_negotiation.cc:407] Connection header received
0927 18:48:27.702116 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:27.702123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:27.702329 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:27.702473 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:27.715791 (+ 13318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.716939 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.718447 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.719249 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.719356 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.719520 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:27.720332 (+ 812us) server_negotiation.cc:299] Negotiation successful
0927 18:48:27.720436 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5017,"thread_start_us":139,"threads_started":1}
I20260927 18:48:27.723594 9575 tablet_bootstrap.cc:492] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e: Bootstrap starting.
I20260927 18:48:27.723775 9490 tablet_service.cc:1467] Processing CreateTablet for tablet 6c206887f6c04aad83eb7d080724a36c (DEFAULT_TABLE table=test-table [id=71e7073c97b94d51a51a9ef61d55f952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:48:27.724818 9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c206887f6c04aad83eb7d080724a36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:48:27.725921 9575 tablet_bootstrap.cc:654] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:27.726755 9575 log.cc:824] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:27.727908 9574 tablet_bootstrap.cc:492] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c: Bootstrap starting.
I20260927 18:48:27.730499 9574 tablet_bootstrap.cc:654] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:27.731247 9574 log.cc:824] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:27.732018 9578 tablet_bootstrap.cc:492] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: Bootstrap starting.
I20260927 18:48:27.734406 9578 tablet_bootstrap.cc:654] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: Neither blocks nor log segments found. Creating new log.
I20260927 18:48:27.734520 9574 tablet_bootstrap.cc:492] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c: No bootstrap required, opened a new log
I20260927 18:48:27.734752 9574 ts_tablet_manager.cc:1397] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:48:27.735498 9578 log.cc:824] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: Log is configured to *not* fsync() on all Append() calls
I20260927 18:48:27.737723 9578 tablet_bootstrap.cc:492] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: No bootstrap required, opened a new log
I20260927 18:48:27.737895 9578 ts_tablet_manager.cc:1397] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:48:27.739215 9574 raft_consensus.cc:348] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 34755 } } peers { permanent_uuid: "d0bb554aa93e46bebf38470509dadc1e" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39105 } } peers { permanent_uuid: "09214af816b847c1a0bc3beb991c2794" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 32875 } }
I20260927 18:48:27.739490 9574 raft_consensus.cc:374] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:27.739542 9574 raft_consensus.cc:729] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f7b1248d864c9b9620d5b2f506e06c, State: Initialized, Role: FOLLOWER
I20260927 18:48:27.740047 9574 consensus_queue.cc:260] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 34755 } } peers { permanent_uuid: "d0bb554aa93e46bebf38470509dadc1e" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39105 } } peers { permanent_uuid: "09214af816b847c1a0bc3beb991c2794" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 32875 } }
I20260927 18:48:27.741020 9574 ts_tablet_manager.cc:1428] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:48:27.741361 9422 heartbeater.cc:503] Master 127.29.159.126:41861 was elected leader, sending a full tablet report...
I20260927 18:48:27.741653 9578 raft_consensus.cc:348] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 34755 } } peers { permanent_uuid: "d0bb554aa93e46bebf38470509dadc1e" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39105 } } peers { permanent_uuid: "09214af816b847c1a0bc3beb991c2794" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 32875 } }
I20260927 18:48:27.741888 9578 raft_consensus.cc:374] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:27.741935 9578 raft_consensus.cc:729] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09214af816b847c1a0bc3beb991c2794, State: Initialized, Role: FOLLOWER
I20260927 18:48:27.742879 9578 consensus_queue.cc:260] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 34755 } } peers { permanent_uuid: "d0bb554aa93e46bebf38470509dadc1e" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39105 } } peers { permanent_uuid: "09214af816b847c1a0bc3beb991c2794" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 32875 } }
I20260927 18:48:27.743940 9578 ts_tablet_manager.cc:1428] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:48:27.744339 9556 heartbeater.cc:503] Master 127.29.159.126:41861 was elected leader, sending a full tablet report...
I20260927 18:48:27.747551 9575 tablet_bootstrap.cc:492] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e: No bootstrap required, opened a new log
I20260927 18:48:27.747767 9575 ts_tablet_manager.cc:1397] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.002s
I20260927 18:48:27.751663 9575 raft_consensus.cc:348] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 34755 } } peers { permanent_uuid: "d0bb554aa93e46bebf38470509dadc1e" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39105 } } peers { permanent_uuid: "09214af816b847c1a0bc3beb991c2794" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 32875 } }
I20260927 18:48:27.751897 9575 raft_consensus.cc:374] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:48:27.751992 9575 raft_consensus.cc:729] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0bb554aa93e46bebf38470509dadc1e, State: Initialized, Role: FOLLOWER
I20260927 18:48:27.752445 9575 consensus_queue.cc:260] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 34755 } } peers { permanent_uuid: "d0bb554aa93e46bebf38470509dadc1e" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39105 } } peers { permanent_uuid: "09214af816b847c1a0bc3beb991c2794" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 32875 } }
I20260927 18:48:27.753113 9286 heartbeater.cc:503] Master 127.29.159.126:41861 was elected leader, sending a full tablet report...
I20260927 18:48:27.753445 9575 ts_tablet_manager.cc:1428] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
W20260927 18:48:27.816051 9423 tablet.cc:2369] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:48:27 dist-test-slave-8tg2 krb5kdc[8596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534907, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
W20260927 18:48:27.886085 9557 tablet.cc:2369] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:48:27.986033 9581 raft_consensus.cc:482] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:48:27.986291 9581 raft_consensus.cc:504] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 34755 } } peers { permanent_uuid: "d0bb554aa93e46bebf38470509dadc1e" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39105 } } peers { permanent_uuid: "09214af816b847c1a0bc3beb991c2794" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 32875 } }
I20260927 18:48:27.987779 9581 leader_election.cc:288] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4f7b1248d864c9b9620d5b2f506e06c (127.29.159.66:34755), d0bb554aa93e46bebf38470509dadc1e (127.29.159.65:39105)
W20260927 18:48:27.987970 9287 tablet.cc:2369] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:48:27.995000 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.988270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:34755 (local address 127.29.159.67:51393)
0927 18:48:27.988493 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:27.988511 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:27.988607 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:27.990901 (+ 2294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:27.990905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:27.990928 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:27.991050 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.991058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.992181 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.992184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.993475 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.993489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.994570 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.994574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.994679 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.994703 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:48:27.994774 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:48:27.994853 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 18:48:27.995968 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.988343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:51393 (local address 127.29.159.66:34755)
0927 18:48:27.990398 (+ 2055us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:27.990403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:27.990420 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:48:27.990475 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:27.990478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:27.990529 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:27.990610 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:27.991191 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.992038 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.993705 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.994329 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.994451 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.994507 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:27.995766 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0927 18:48:27.995850 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355}
I20260927 18:48:27.999383 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.988574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:39105 (local address 127.29.159.67:48025)
0927 18:48:27.995175 (+ 6601us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:27.995190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:48:27.995281 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:48:27.995655 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:48:27.995658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:48:27.995675 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:48:27.995756 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.995763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.996839 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.996843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.997975 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:48:27.997986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.998975 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.998978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:48:27.999061 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.999081 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:48:27.999147 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:48:27.999210 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6541,"thread_start_us":121,"threads_started":1}
I20260927 18:48:27.999826 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:27.990786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:48025 (local address 127.29.159.65:39105)
0927 18:48:27.990919 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:27.990924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:27.995267 (+ 4343us) server_negotiation.cc:407] Connection header received
0927 18:48:27.995385 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:27.995388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:27.995464 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:27.995549 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:48:27.995883 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.996666 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.998114 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:27.998740 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:27.998834 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:27.998882 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:27.999636 (+ 754us) server_negotiation.cc:299] Negotiation successful
0927 18:48:27.999712 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:48:28.000335 9365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c206887f6c04aad83eb7d080724a36c" candidate_uuid: "09214af816b847c1a0bc3beb991c2794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" is_pre_election: true
I20260927 18:48:28.000510 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c206887f6c04aad83eb7d080724a36c" candidate_uuid: "09214af816b847c1a0bc3beb991c2794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bb554aa93e46bebf38470509dadc1e" is_pre_election: true
I20260927 18:48:28.000665 9365 raft_consensus.cc:2454] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09214af816b847c1a0bc3beb991c2794 in term 0.
I20260927 18:48:28.000780 9224 raft_consensus.cc:2454] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09214af816b847c1a0bc3beb991c2794 in term 0.
I20260927 18:48:28.001451 9446 leader_election.cc:302] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09214af816b847c1a0bc3beb991c2794, e4f7b1248d864c9b9620d5b2f506e06c; no voters:
I20260927 18:48:28.001770 9581 raft_consensus.cc:2781] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:48:28.001852 9581 raft_consensus.cc:482] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:48:28.001892 9581 raft_consensus.cc:3037] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:28.003290 9581 raft_consensus.cc:504] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 34755 } } peers { permanent_uuid: "d0bb554aa93e46bebf38470509dadc1e" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39105 } } peers { permanent_uuid: "09214af816b847c1a0bc3beb991c2794" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 32875 } }
I20260927 18:48:28.004379 9581 leader_election.cc:288] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [CANDIDATE]: Term 1 election: Requested vote from peers e4f7b1248d864c9b9620d5b2f506e06c (127.29.159.66:34755), d0bb554aa93e46bebf38470509dadc1e (127.29.159.65:39105)
I20260927 18:48:28.005093 9365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c206887f6c04aad83eb7d080724a36c" candidate_uuid: "09214af816b847c1a0bc3beb991c2794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f7b1248d864c9b9620d5b2f506e06c"
I20260927 18:48:28.005223 9365 raft_consensus.cc:3037] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:28.006527 9365 raft_consensus.cc:2454] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09214af816b847c1a0bc3beb991c2794 in term 1.
I20260927 18:48:28.006940 9446 leader_election.cc:302] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09214af816b847c1a0bc3beb991c2794, e4f7b1248d864c9b9620d5b2f506e06c; no voters:
I20260927 18:48:28.007229 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c206887f6c04aad83eb7d080724a36c" candidate_uuid: "09214af816b847c1a0bc3beb991c2794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bb554aa93e46bebf38470509dadc1e"
I20260927 18:48:28.007368 9224 raft_consensus.cc:3037] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:48:28.008673 9224 raft_consensus.cc:2454] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09214af816b847c1a0bc3beb991c2794 in term 1.
I20260927 18:48:28.009217 9581 raft_consensus.cc:2781] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:48:28.009449 9581 raft_consensus.cc:686] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 1 LEADER]: Becoming Leader. State: Replica: 09214af816b847c1a0bc3beb991c2794, State: Running, Role: LEADER
I20260927 18:48:28.009814 9581 consensus_queue.cc:237] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 34755 } } peers { permanent_uuid: "d0bb554aa93e46bebf38470509dadc1e" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39105 } } peers { permanent_uuid: "09214af816b847c1a0bc3beb991c2794" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 32875 } }
I20260927 18:48:28.017434 9072 catalog_manager.cc:5495] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09214af816b847c1a0bc3beb991c2794 (127.29.159.67). New cstate: current_term: 1 leader_uuid: "09214af816b847c1a0bc3beb991c2794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0bb554aa93e46bebf38470509dadc1e" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09214af816b847c1a0bc3beb991c2794" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 32875 } health_report { overall_health: HEALTHY } } }
I20260927 18:48:28.074867 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:28.069704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.29.159.67:32875)
0927 18:48:28.069844 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 18:48:28.069849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:48:28.069864 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:48:28.069954 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:48:28.069958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:48:28.070026 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:48:28.070135 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:48:28.070545 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:28.071333 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:28.072602 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:48:28.072950 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:48:28.073037 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:48:28.073429 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:48:28.074076 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:48:28.074200 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:48:28.074625 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 18:48:28.074687 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:48:28.082463 9224 raft_consensus.cc:1261] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e [term 1 FOLLOWER]: Refusing update from remote peer 09214af816b847c1a0bc3beb991c2794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:48:28.083393 9586 consensus_queue.cc:1035] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0bb554aa93e46bebf38470509dadc1e" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:48:28.083745 9365 raft_consensus.cc:1261] T 6c206887f6c04aad83eb7d080724a36c P e4f7b1248d864c9b9620d5b2f506e06c [term 1 FOLLOWER]: Refusing update from remote peer 09214af816b847c1a0bc3beb991c2794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:48:28.084406 9586 consensus_queue.cc:1035] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f7b1248d864c9b9620d5b2f506e06c" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:48:28.090543 9592 mvcc.cc:205] Tried to move back new op lower bound from 7334030983496871936 to 7334030983211708416. Current Snapshot: MvccSnapshot[applied={T|T < 7334030983496871936}]
I20260927 18:48:28.093885 9597 mvcc.cc:205] Tried to move back new op lower bound from 7334030983496871936 to 7334030983211708416. Current Snapshot: MvccSnapshot[applied={T|T < 7334030983496871936}]
I20260927 18:48:28.097041 9598 mvcc.cc:205] Tried to move back new op lower bound from 7334030983496871936 to 7334030983211708416. Current Snapshot: MvccSnapshot[applied={T|T < 7334030983496871936}]
I20260927 18:48:28.103892 9072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:48:28.104055 9072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:48:28.106470 9072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1f91c425cdc47b3b539bc8a537b097c: Sending DeleteTablet for 3 replicas of tablet 6c206887f6c04aad83eb7d080724a36c
I20260927 18:48:28.107208 9356 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c206887f6c04aad83eb7d080724a36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:28 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:33802
I20260927 18:48:28.107486 9220 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c206887f6c04aad83eb7d080724a36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:28 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:43710
I20260927 18:48:28.107590 9490 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c206887f6c04aad83eb7d080724a36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:48:28 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:59516
I20260927 18:48:28.107959 9606 tablet_replica.cc:330] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e: stopping tablet replica
I20260927 18:48:28.108269 9606 raft_consensus.cc:2229] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:48:28.108618 9606 raft_consensus.cc:2258] T 6c206887f6c04aad83eb7d080724a36c P d0bb554aa93e46bebf38470509dadc1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:28.108770 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 9153
W20260927 18:48:28.119167 9057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:39105 (error 108)
I20260927 18:48:28.119627 9607 tablet_replica.cc:330] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: stopping tablet replica
I20260927 18:48:28.119941 9607 raft_consensus.cc:2229] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:48:28.120195 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 9292
I20260927 18:48:28.120287 9607 raft_consensus.cc:2258] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:48:28.122344 9607 ts_tablet_manager.cc:1905] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:48:28.124218 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:28.123919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:39105 (local address 127.0.0.1:43714)
0927 18:48:28.124060 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:28.124134 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:39105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 18:48:28.124382 9057 catalog_manager.cc:4584] TS d0bb554aa93e46bebf38470509dadc1e (127.29.159.65:39105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c206887f6c04aad83eb7d080724a36c: Network error: Client connection negotiation failed: client connection to 127.29.159.65:39105: connect: Connection refused (error 111)
I20260927 18:48:28.130321 9607 ts_tablet_manager.cc:1918] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:48:28.130478 9607 log.cc:1196] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/6c206887f6c04aad83eb7d080724a36c
I20260927 18:48:28.130887 9607 ts_tablet_manager.cc:1939] T 6c206887f6c04aad83eb7d080724a36c P 09214af816b847c1a0bc3beb991c2794: Deleting consensus metadata
I20260927 18:48:28.132189 9057 catalog_manager.cc:4841] TS 09214af816b847c1a0bc3beb991c2794 (127.29.159.67:32875): tablet 6c206887f6c04aad83eb7d080724a36c (table test-table [id=71e7073c97b94d51a51a9ef61d55f952]) successfully deleted
I20260927 18:48:28.133210 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:48:28.132900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:34755 (local address 127.0.0.1:33818)
0927 18:48:28.133061 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:48:28.133115 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.66:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 18:48:28.133435 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 9427
I20260927 18:48:28.140043 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 9040
I20260927 18:48:28.148905 30333 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:48:28.493796 30333 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:48:28.493973 30333 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:48:28.838382 30333 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:48:28.838 UTC [8626] LOG: received smart shutdown request
2026-09-27 18:48:28.841 UTC [8626] LOG: background worker "logical replication launcher" (PID 8635) exited with exit code 1
2026-09-27 18:48:28.842 UTC [8630] LOG: shutting down
2026-09-27 18:48:29.010 UTC [8626] LOG: database system is shut down
2026-09-27T18:48:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45716 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:48:29 dist-test-slave-8tg2 krb5kdc[9617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:48:29 dist-test-slave-8tg2 krb5kdc[9617](info): set up 2 sockets
Sep 27 18:48:29 dist-test-slave-8tg2 krb5kdc[9617](info): commencing operation
krb5kdc: starting...
W20260927 18:48:30.167620 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:48:30 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:48:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:48:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.159.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.159.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/rangeradmin_127.29.159.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/rangeradmin_127.29.159.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.159.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.159.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/rangerlookup_127.29.159.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/rangerlookup_127.29.159.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.159.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.159.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/HTTP_127.29.159.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/HTTP_127.29.159.84@KRBTEST.COM.keytab.
I20260927 18:48:30.337553 30333 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskclkeqe/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/postgres -l logfile start
2026-09-27 18:48:32.824 UTC [9647] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:48:32.824 UTC [9647] LOG: listening on IPv4 address "127.29.159.84", port 42747
2026-09-27 18:48:32.827 UTC [9647] LOG: listening on Unix socket "/tmp/.s.PGSQL.42747"
2026-09-27 18:48:32.848 UTC [9650] LOG: database system was shut down at 2026-09-27 18:48:31 UTC
2026-09-27 18:48:32.852 UTC [9647] LOG: database system is ready to accept connections
I20260927 18:48:33.815491 30333 mini_postgres.cc:96] Postgres bound to 42747
2026-09-27 18:48:33.821 UTC [9658] FATAL: database "slave" does not exist
127.29.159.84:42747 - accepting connections
I20260927 18:48:33.822357 30333 mini_ranger.cc:161] Starting Ranger...
I20260927 18:48:33.837267 30333 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:48:34.325557 30333 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:48:34.325632 30333 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-admin
2026-09-27 18:48:34,597 [I] DB FLAVOR :POSTGRES
2026-09-27 18:48:34,597 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:48:34,598 [I] Checking connection..
2026-09-27 18:48:34,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:48:34,860 [I] Checking connection passed.
2026-09-27 18:48:34,861 [I] --------- Verifying version history table ---------
2026-09-27 18:48:34,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:48:35,111 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:48:35,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:48:35,352 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:48:35,353 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:48:35,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:48:35.578 UTC [9722] WARNING: there is no transaction in progress
2026-09-27 18:48:35,584 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:48:35,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:48:35,822 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:48:35,823 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:48:35,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:48:36,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:48:36,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8tg2.c.gcp-upstream.internal','N') ;"
2026-09-27 18:48:36,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:48:36,767 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:48:36,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:48:37,016 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:48:37,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:48:37,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:48:37,500 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:48:37,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:48:38.483 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.500 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.516 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.531 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.547 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.565 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.625 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.632 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.646 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.656 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.667 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.681 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.691 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.697 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.704 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:38.711 UTC [9830] WARNING: there is no transaction in progress
2026-09-27 18:48:39,357 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:48:39,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:48:39,611 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:48:39,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:48:39,859 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:48:39,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:48:40,087 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:48:40,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:48:40,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:48:40,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8tg2.c.gcp-upstream.internal';"
2026-09-27 18:48:40,803 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:48:40,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:48:41,046 [I] DB_PATCHES have already been applied
I20260927 18:48:41.052233 30333 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-admin:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskclkeqe/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:48:41.052295 30333 mini_ranger.cc:193] Using host: 127.29.159.84
I20260927 18:48:41.055624 30333 mini_ranger.cc:227] Ranger admin URL: http://127.29.159.84:54843
Sep 27, 2026 6:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:48:42.153728 30333 mini_ranger.cc:160] Time spent starting Ranger: real 8.331s user 0.000s sys 0.009s
Sep 27, 2026 6:49:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:49:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 6:49:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:49:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:49:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:49:04.290745 30333 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.159.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.159.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/rangerkms_127.29.159.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.159.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/rangerkms_127.29.159.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.159.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.159.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.159.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/HTTP_127.29.159.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.159.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/HTTP_127.29.159.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:49:04 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534944, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:49:04.380298 30333 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:49:04.389842 30333 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:49:05.258452 30333 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:49:05.258544 30333 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-kms
I20260927 18:49:05.258567 30333 mini_ranger_kms.cc:227] Using postgres at 127.29.159.84:42747
2026-09-27 18:49:05,428 [I] DB FLAVOR :POSTGRES
2026-09-27 18:49:05,428 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:49:05,428 [I] Checking connection
2026-09-27 18:49:05,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:49:05,659 [I] connection success
2026-09-27 18:49:05,660 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:49:05,660 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:49:05,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:49:05,907 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:49:05,907 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:49:05,908 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:49:05,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.159.84:42747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:49:06,177 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:49:06.660565 30333 mini_ranger_kms.cc:310] Created kms service
I20260927 18:49:06.865928 30333 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:49:06.909682 30333 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:49:06 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534944, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:49:07.325804 30333 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:49:07.325961 30333 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-kms:/tmp/dist-test-taskclkeqe/build/debug/bin/postgresql.jar:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskclkeqe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:49:07.325984 30333 mini_ranger_kms.cc:240] Using host: 127.29.159.84
I20260927 18:49:07.330160 30333 mini_ranger_kms.cc:276] Ranger KMS PID: 10010
I20260927 18:49:07.330278 30333 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.159.84:58117
Sep 27, 2026 6:49:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:49:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:49:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:49:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:49:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:49:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 18:49:09.597424 30333 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.217s user 0.016s sys 0.137s
I20260927 18:49:09.597559 30333 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:49:09.597657 30333 mini_ranger_kms.cc:401] 127.29.159.84:58117/kms/v1/keys
Sep 27 18:49:09 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534949, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.159.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 18:49:10 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534949, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.159.84@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
Sep 27 18:49:10 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534949, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.159.84@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
Sep 27 18:49:12 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:49:12.871570 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41721
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:41721
--ranger_config_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:58117/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:12.981863 10061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:12.982122 10061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:12.982161 10061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:12.986136 10061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:12.986197 10061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:12.986217 10061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:12.986232 10061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:12.986248 10061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:12.991788 10061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.159.84:58117/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:41721
--ranger_config_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:12.992754 10061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:12.993763 10061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:49:13.002628 10061 server_base.cc:1034] running on GCE node
W20260927 18:49:13.002560 10066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:13.002645 10067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:13.002578 10069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:13.003401 10061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:13.004563 10061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:13.005719 10061 hybrid_clock.cc:648] HybridClock initialized: now 1790534953005696 us; error 38 us; skew 500 ppm
Sep 27 18:49:13 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534953, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:13.008489 10061 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:49:13.009369 10061 webserver.cc:460] Webserver started at http://127.29.159.126:46747/ using document root <none> and password file <none>
I20260927 18:49:13.009933 10061 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:13.010015 10061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:13.010244 10061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:49:13 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534953, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:49:13.068539 10061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "da8b270621eb438091ce19ec34e04ed2"
format_stamp: "Formatted at 2026-09-27 18:49:13 on dist-test-slave-8tg2"
server_key: "4c7f384fca809943a46f5d0a4146195c"
server_key_iv: "b08fcbb07d021e63cba2efb4a02a8835"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:13.069332 10061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "da8b270621eb438091ce19ec34e04ed2"
format_stamp: "Formatted at 2026-09-27 18:49:13 on dist-test-slave-8tg2"
server_key: "4c7f384fca809943a46f5d0a4146195c"
server_key_iv: "b08fcbb07d021e63cba2efb4a02a8835"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:13.074504 10061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:49:13.089219 10076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:13.090736 10061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:49:13.090878 10061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "da8b270621eb438091ce19ec34e04ed2"
format_stamp: "Formatted at 2026-09-27 18:49:13 on dist-test-slave-8tg2"
server_key: "4c7f384fca809943a46f5d0a4146195c"
server_key_iv: "b08fcbb07d021e63cba2efb4a02a8835"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:13.091012 10061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:13.112201 10061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:13.115294 10061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:13.115574 10061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:13.123960 10061 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:41721
I20260927 18:49:13.124007 10128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:41721 every 8 connection(s)
I20260927 18:49:13.124979 10061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:13.128480 10129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:13.129019 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 10061
I20260927 18:49:13.129153 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:49:13.129480 30333 external_mini_cluster.cc:1428] Setting key 66551265e0aab3698e4577206b6c3376
I20260927 18:49:13.135876 10129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2: Bootstrap starting.
Sep 27 18:49:13 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:13.138661 10129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:13.139611 10129 log.cc:824] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:13.141927 10129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2: No bootstrap required, opened a new log
I20260927 18:49:13.144342 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:13.130955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.29.159.126:41721)
0927 18:49:13.131400 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:13.131438 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:13.131693 (+ 255us) server_negotiation.cc:407] Connection header received
0927 18:49:13.132494 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:13.132513 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:13.132861 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:13.133207 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:13.134013 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:13.134996 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:13.135977 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:13.136451 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:13.136621 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:13.139141 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:13.139147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:13.139159 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:13.139185 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:13.141571 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:13.142030 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:13.142036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:13.142043 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:13.142162 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:13.142448 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:13.142453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:13.142455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:13.142840 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:13.143046 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:13.143313 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 18:49:13.143636 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":167,"threads_started":1}
I20260927 18:49:13.145799 10129 raft_consensus.cc:348] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } }
I20260927 18:49:13.146064 10129 raft_consensus.cc:374] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:13.146144 10129 raft_consensus.cc:729] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8b270621eb438091ce19ec34e04ed2, State: Initialized, Role: FOLLOWER
I20260927 18:49:13.146657 10129 consensus_queue.cc:260] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } }
I20260927 18:49:13.146818 10129 raft_consensus.cc:388] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:13.146879 10129 raft_consensus.cc:482] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:13.146971 10129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:13.148109 10129 raft_consensus.cc:504] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } }
I20260927 18:49:13.148517 10129 leader_election.cc:302] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da8b270621eb438091ce19ec34e04ed2; no voters:
I20260927 18:49:13.148860 10129 leader_election.cc:288] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:49:13.148967 10134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:13.149238 10134 raft_consensus.cc:686] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 1 LEADER]: Becoming Leader. State: Replica: da8b270621eb438091ce19ec34e04ed2, State: Running, Role: LEADER
I20260927 18:49:13.149574 10134 consensus_queue.cc:237] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } }
I20260927 18:49:13.150090 10129 sys_catalog.cc:564] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:13.151173 10135 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da8b270621eb438091ce19ec34e04ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } } }
I20260927 18:49:13.151299 10135 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:13.151684 10136 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da8b270621eb438091ce19ec34e04ed2. Latest consensus state: current_term: 1 leader_uuid: "da8b270621eb438091ce19ec34e04ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } } }
I20260927 18:49:13.151820 10136 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:13.152241 10129 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:49:13.153390 10143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:13.160657 10143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:13.168164 10143 catalog_manager.cc:1347] Generated new cluster ID: 1446410deed54cd998b3ab7dfea6445c
I20260927 18:49:13.168220 10143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:13.212698 10143 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:13.215283 10143 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:13.225502 10143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2: Generated new TSK 0
I20260927 18:49:13.226528 10143 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:49:13.883136 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:58117/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41721
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:14.003407 10174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:14.003774 10174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:14.003895 10174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:14.008308 10174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:14.008394 10174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:14.008540 10174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:49:14.014582 10174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.159.84:58117/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:14.015731 10174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:14.016775 10174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:14.027303 10179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:14.027267 10180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:14.027325 10182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:14.027997 10174 server_base.cc:1034] running on GCE node
I20260927 18:49:14.028342 10174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:14.028936 10174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:14.030107 10174 hybrid_clock.cc:648] HybridClock initialized: now 1790534954030087 us; error 33 us; skew 500 ppm
Sep 27 18:49:14 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534954, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:14.032915 10174 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:49:14.033860 10174 webserver.cc:460] Webserver started at http://127.29.159.65:34499/ using document root <none> and password file <none>
I20260927 18:49:14.034490 10174 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:14.034579 10174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:14.034799 10174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:49:14 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534954, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:49:14.052603 10174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "abf1a5cb5802426dae1e379464dc78d8"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "7fd90da3d051da381d8aa130808e1624"
server_key_iv: "40660ce580153d23d168cb16edf66b1f"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:14.053333 10174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "abf1a5cb5802426dae1e379464dc78d8"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "7fd90da3d051da381d8aa130808e1624"
server_key_iv: "40660ce580153d23d168cb16edf66b1f"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:14.058393 10174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:49:14.078040 10191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:14.079895 10174 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260927 18:49:14.080035 10174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "abf1a5cb5802426dae1e379464dc78d8"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "7fd90da3d051da381d8aa130808e1624"
server_key_iv: "40660ce580153d23d168cb16edf66b1f"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:14.080180 10174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:14.104453 10174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:14.107998 10174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:14.108227 10174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:14.108925 10174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:14.109952 10174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:14.110040 10174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:14.110108 10174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:14.110145 10174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:14.119828 10174 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:33755
I20260927 18:49:14.121085 10174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:14.122624 10304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:33755 every 8 connection(s)
I20260927 18:49:14.131289 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 10174
I20260927 18:49:14.131650 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:49:14.132037 30333 external_mini_cluster.cc:1428] Setting key 55f32789fa7bf01237a08b1aaaa43c0e
Sep 27 18:49:14 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534954, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:14.137845 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:14.124203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:48815 (local address 127.29.159.126:41721)
0927 18:49:14.125646 (+ 1443us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:14.125656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:14.125675 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:49:14.126230 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:14.126234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:14.126293 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:14.126387 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:14.127477 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.128273 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.129802 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.130166 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.130253 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:14.133402 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:14.133406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:14.133409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:14.133434 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:14.135447 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:14.136028 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:14.136032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:14.136034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:14.136116 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:14.136483 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:14.136486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:14.136488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:14.136681 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:14.136769 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:14.137561 (+ 792us) server_negotiation.cc:299] Negotiation successful
0927 18:49:14.137685 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1363,"thread_start_us":104,"threads_started":1}
I20260927 18:49:14.139541 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:14.124638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41721 (local address 127.29.159.65:48815)
0927 18:49:14.125094 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:14.125145 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:14.126019 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:14.126569 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:14.126575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:14.126836 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:14.127295 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:14.127307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.128414 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.128417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:14.129670 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:14.129679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.130412 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.130416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:14.130595 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:14.131264 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:14.131287 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:14.133154 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:14.135586 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:14.135592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:14.135607 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:14.135920 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:14.136221 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:14.136224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:14.136226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:14.136378 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:14.136829 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:14.136836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:14.137142 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 18:49:14.137348 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 18:49:14.138572 (+ 1224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":137,"threads_started":1}
I20260927 18:49:14.141333 10305 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41721
I20260927 18:49:14.141563 10305 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:14.142068 10305 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
I20260927 18:49:14.143841 10093 ts_manager.cc:194] Registered new tserver with Master: abf1a5cb5802426dae1e379464dc78d8 (127.29.159.65:33755)
I20260927 18:49:14.145344 10093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:48815
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:49:14.186444 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:58117/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41721
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:49:14 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534954, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:49:14.311488 10313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:14.311815 10313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:14.311928 10313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:14.316524 10313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:14.316596 10313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:14.316689 10313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:49:14.321941 10313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.159.84:58117/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:14.322959 10313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:14.324047 10313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:14.333600 10320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:14.333577 10322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:14.333817 10313 server_base.cc:1034] running on GCE node
W20260927 18:49:14.334097 10319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:14.334522 10313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:14.335294 10313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:14.336485 10313 hybrid_clock.cc:648] HybridClock initialized: now 1790534954336455 us; error 45 us; skew 500 ppm
Sep 27 18:49:14 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534954, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:14.339612 10313 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:49:14.340754 10313 webserver.cc:460] Webserver started at http://127.29.159.66:43039/ using document root <none> and password file <none>
I20260927 18:49:14.341493 10313 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:14.341636 10313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:14.341928 10313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:49:14 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534954, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:49:14.358466 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "47e7a43ef9d94b16916ec65c330e4529"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "d5b3a0d72f8f3f37bcd5b9da645a5021"
server_key_iv: "8201f95fef8217a2d11e7507d82e7640"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:14.359302 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "47e7a43ef9d94b16916ec65c330e4529"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "d5b3a0d72f8f3f37bcd5b9da645a5021"
server_key_iv: "8201f95fef8217a2d11e7507d82e7640"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:14.364506 10313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 18:49:14.380512 10329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:14.382355 10313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:49:14.382493 10313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "47e7a43ef9d94b16916ec65c330e4529"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "d5b3a0d72f8f3f37bcd5b9da645a5021"
server_key_iv: "8201f95fef8217a2d11e7507d82e7640"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:14.382700 10313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:14.412081 10313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:14.415993 10313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:14.416241 10313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:14.417013 10313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:14.418119 10313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:14.418246 10313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:14.418345 10313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:14.418416 10313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:14.431891 10313 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:37469
I20260927 18:49:14.433115 10313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:14.434490 10442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:37469 every 8 connection(s)
I20260927 18:49:14.437024 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 10313
I20260927 18:49:14.437155 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:49:14.437449 30333 external_mini_cluster.cc:1428] Setting key ff998afd05a5151d96ff93f04e707a0b
Sep 27 18:49:14 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534954, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:14.459764 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:14.436774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:33855 (local address 127.29.159.126:41721)
0927 18:49:14.436926 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:14.436931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:14.443934 (+ 7003us) server_negotiation.cc:407] Connection header received
0927 18:49:14.445180 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:14.445183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:14.445235 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:14.445286 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:14.446883 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.447696 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.449391 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.449762 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.449840 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:14.455453 (+ 5613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:14.455458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:14.455461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:14.455488 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:14.457259 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:14.457915 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:14.457919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:14.457921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:14.458005 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:14.458421 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:14.458425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:14.458427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:14.458656 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:14.458739 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:14.459442 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 18:49:14.459584 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:49:14.461659 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:14.437333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41721 (local address 127.29.159.66:33855)
0927 18:49:14.443773 (+ 6440us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:14.443834 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:14.444928 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:14.445695 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:14.445716 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:14.446064 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:14.446719 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:14.446738 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.447870 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.447873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:14.449253 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:14.449264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.450043 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.450047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:14.450236 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:14.453366 (+ 3130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:14.453386 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:14.455223 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:14.457405 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:14.457411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:14.457430 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:14.457793 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:14.458111 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:14.458114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:14.458116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:14.458308 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:14.458824 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:14.458831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:14.459142 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 18:49:14.460225 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0927 18:49:14.460609 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6189,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:49:14.468492 10443 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41721
I20260927 18:49:14.468776 10443 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:14.469376 10443 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
I20260927 18:49:14.470546 10093 ts_manager.cc:194] Registered new tserver with Master: 47e7a43ef9d94b16916ec65c330e4529 (127.29.159.66:37469)
I20260927 18:49:14.471304 10093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:33855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:49:14.495121 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:58117/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41721
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:14.656847 10448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:14.657123 10448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:14.657207 10448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:14.661476 10448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:14.661563 10448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:14.661669 10448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:49:14.667807 10448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.159.84:58117/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:14.669251 10448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:14.670584 10448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:14.684255 10454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:14.684612 10453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:14.685312 10456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:14.690722 10448 server_base.cc:1034] running on GCE node
I20260927 18:49:14.691195 10448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:14.692237 10448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:14.699574 10448 hybrid_clock.cc:648] HybridClock initialized: now 1790534954699487 us; error 90 us; skew 500 ppm
Sep 27 18:49:14 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534954, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
Sep 27 18:49:14 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534954, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:14.703920 10448 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:49:14.705107 10448 webserver.cc:460] Webserver started at http://127.29.159.67:39555/ using document root <none> and password file <none>
I20260927 18:49:14.705900 10448 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:14.706053 10448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:14.706363 10448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:49:14 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534954, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:49:14.735172 10448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "c3c2c2020a774358a3751464155fde77"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "c6546fd1962b291dfc3490ff08dc84a5"
server_key_iv: "5dda7009b48b3432b7da3601f44de8a8"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:14.741902 10448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "c3c2c2020a774358a3751464155fde77"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "c6546fd1962b291dfc3490ff08dc84a5"
server_key_iv: "5dda7009b48b3432b7da3601f44de8a8"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:14.761850 10448 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.004s sys 0.000s
I20260927 18:49:14.780102 10463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:14.781860 10448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:49:14.781989 10448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "c3c2c2020a774358a3751464155fde77"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "c6546fd1962b291dfc3490ff08dc84a5"
server_key_iv: "5dda7009b48b3432b7da3601f44de8a8"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:14.782115 10448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:14.813629 10448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:14.818163 10448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:14.818390 10448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:14.819079 10448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:14.820103 10448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:14.820176 10448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:14.820284 10448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:14.820331 10448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:14.831319 10448 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:42741
I20260927 18:49:14.832460 10448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:14.831342 10576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:42741 every 8 connection(s)
I20260927 18:49:14.838320 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 10448
I20260927 18:49:14.838436 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:49:14.838836 30333 external_mini_cluster.cc:1428] Setting key ec7e45fbbc010337d61ebad522f6ae8f
Sep 27 18:49:14 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534954, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:14.848855 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:14.834315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:51715 (local address 127.29.159.126:41721)
0927 18:49:14.834445 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:14.834449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:14.835449 (+ 1000us) server_negotiation.cc:407] Connection header received
0927 18:49:14.836478 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:14.836481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:14.836539 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:14.836666 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:14.838731 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.839586 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.841337 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.841685 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.841813 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:14.844682 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:14.844686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:14.844689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:14.844717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:14.846476 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:14.847065 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:14.847069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:14.847071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:14.847172 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:14.847581 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:14.847585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:14.847587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:14.847825 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:14.847927 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:14.848547 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 18:49:14.848672 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:49:14.849749 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:14.834802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41721 (local address 127.29.159.67:51715)
0927 18:49:14.835271 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:14.835323 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:14.836218 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:14.836880 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:14.836889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:14.837239 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:14.837900 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:14.837921 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.839754 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.839758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:14.841140 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:14.841151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:14.841851 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:14.841855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:14.842047 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:14.842757 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:14.842776 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:14.844505 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:14.846607 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:14.846614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:14.846629 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:14.846952 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:14.847279 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:14.847282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:14.847284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:14.847481 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:14.847936 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:14.847943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:14.848251 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 18:49:14.848443 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 18:49:14.848755 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":143,"threads_started":1}
I20260927 18:49:14.850931 10577 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41721
I20260927 18:49:14.851223 10577 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:14.851900 10577 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
I20260927 18:49:14.853559 10093 ts_manager.cc:194] Registered new tserver with Master: c3c2c2020a774358a3751464155fde77 (127.29.159.67:42741)
I20260927 18:49:14.854317 10093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:51715
I20260927 18:49:14.857228 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:49:14.857911 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 10174
I20260927 18:49:14.865559 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 10313
I20260927 18:49:14.872485 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 10448
I20260927 18:49:14.880461 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 10061
I20260927 18:49:14.889573 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41721
--webserver_interface=127.29.159.126
--webserver_port=46747
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:41721
--ranger_config_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:58117/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.159.84:58117/kms with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:15.003526 10580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:15.003865 10580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:15.003950 10580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:15.008185 10580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:15.008273 10580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:15.008312 10580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:15.008364 10580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:15.008410 10580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:15.013868 10580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.159.84:58117/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:41721
--ranger_config_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=46747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:15.014969 10580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:15.016083 10580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:15.024694 10588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:15.024749 10585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:15.024914 10580 server_base.cc:1034] running on GCE node
W20260927 18:49:15.025163 10586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:15.025704 10580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:15.026834 10580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:15.028009 10580 hybrid_clock.cc:648] HybridClock initialized: now 1790534955027969 us; error 48 us; skew 500 ppm
Sep 27 18:49:15 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:15.030941 10580 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:49:15.031909 10580 webserver.cc:460] Webserver started at http://127.29.159.126:46747/ using document root <none> and password file <none>
I20260927 18:49:15.032477 10580 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:15.032542 10580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:49:15 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:49:15.098693 10580 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:49:15.101550 10596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:15.102905 10580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:49:15.103035 10580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "da8b270621eb438091ce19ec34e04ed2"
format_stamp: "Formatted at 2026-09-27 18:49:13 on dist-test-slave-8tg2"
server_key: "4c7f384fca809943a46f5d0a4146195c"
server_key_iv: "b08fcbb07d021e63cba2efb4a02a8835"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:15.103654 10580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:15.126240 10580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:15.129421 10580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:15.129580 10580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:15.137092 10580 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:41721
I20260927 18:49:15.137122 10648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:41721 every 8 connection(s)
I20260927 18:49:15.138186 10580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:15.143014 10649 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:49:15.145025 10649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2: Bootstrap starting.
I20260927 18:49:15.147192 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 10580
I20260927 18:49:15.147715 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:58117/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:33755
--local_ip_for_outbound_sockets=127.29.159.65
--tserver_master_addrs=127.29.159.126:41721
--webserver_port=34499
--webserver_interface=127.29.159.65
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.159.84:58117/kms with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:15.166708 10649 log.cc:824] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:15.174899 10649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:49:15.175331 10649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2: Bootstrap complete.
I20260927 18:49:15.180678 10649 raft_consensus.cc:348] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } }
I20260927 18:49:15.181166 10649 raft_consensus.cc:729] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8b270621eb438091ce19ec34e04ed2, State: Initialized, Role: FOLLOWER
I20260927 18:49:15.181705 10649 consensus_queue.cc:260] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } }
I20260927 18:49:15.181847 10649 raft_consensus.cc:388] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:15.181916 10649 raft_consensus.cc:482] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:15.182005 10649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:49:15.184420 10649 raft_consensus.cc:504] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } }
I20260927 18:49:15.184767 10649 leader_election.cc:302] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da8b270621eb438091ce19ec34e04ed2; no voters:
I20260927 18:49:15.185074 10649 leader_election.cc:288] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:49:15.185209 10653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:49:15.185467 10653 raft_consensus.cc:686] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [term 2 LEADER]: Becoming Leader. State: Replica: da8b270621eb438091ce19ec34e04ed2, State: Running, Role: LEADER
I20260927 18:49:15.185760 10653 consensus_queue.cc:237] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } }
I20260927 18:49:15.186233 10649 sys_catalog.cc:564] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:15.186659 10655 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da8b270621eb438091ce19ec34e04ed2. Latest consensus state: current_term: 2 leader_uuid: "da8b270621eb438091ce19ec34e04ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } } }
I20260927 18:49:15.186775 10655 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:15.187074 10654 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "da8b270621eb438091ce19ec34e04ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8b270621eb438091ce19ec34e04ed2" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41721 } } }
I20260927 18:49:15.187143 10654 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:15.187166 10657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:15.190359 10657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:15.190932 10649 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:49:15.191004 10657 catalog_manager.cc:1259] Loaded cluster ID: 1446410deed54cd998b3ab7dfea6445c
I20260927 18:49:15.191049 10657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:15.192893 10657 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:15.193567 10657 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2: Loaded TSK: 0
I20260927 18:49:15.194571 10657 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:49:15.309480 10651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:15.309865 10651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:15.309938 10651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:15.316094 10651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:15.316169 10651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:15.316262 10651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:49:15.324389 10651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.159.84:58117/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:33755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=34499
--tserver_master_addrs=127.29.159.126:41721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:15.325762 10651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:15.327112 10651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:15.339350 10687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:15.339393 10689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:15.339382 10686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:15.340214 10651 server_base.cc:1034] running on GCE node
I20260927 18:49:15.340652 10651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:15.341388 10651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:15.342578 10651 hybrid_clock.cc:648] HybridClock initialized: now 1790534955342552 us; error 53 us; skew 500 ppm
Sep 27 18:49:15 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:15.346028 10651 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:49:15.347020 10651 webserver.cc:460] Webserver started at http://127.29.159.65:34499/ using document root <none> and password file <none>
I20260927 18:49:15.347954 10651 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:15.348060 10651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:49:15 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:49:15.369001 10651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:15.371546 10697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:15.372901 10651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:15.373037 10651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "abf1a5cb5802426dae1e379464dc78d8"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "7fd90da3d051da381d8aa130808e1624"
server_key_iv: "40660ce580153d23d168cb16edf66b1f"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:15.373569 10651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:15.392362 10651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:15.395655 10651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:15.395853 10651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:15.396478 10651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:15.397469 10651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:15.397539 10651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:15.397605 10651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:15.397632 10651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:15.410322 10651 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:33755
I20260927 18:49:15.410413 10810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:33755 every 8 connection(s)
I20260927 18:49:15.411574 10651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:15.417282 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 10651
I20260927 18:49:15.417714 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:58117/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:37469
--local_ip_for_outbound_sockets=127.29.159.66
--tserver_master_addrs=127.29.159.126:41721
--webserver_port=43039
--webserver_interface=127.29.159.66
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.159.84:58117/kms with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:49:15 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:15.434818 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:15.414573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41721 (local address 127.29.159.65:45587)
0927 18:49:15.415348 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:15.415402 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:15.416254 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:15.419865 (+ 3611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:15.419877 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:15.420257 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:15.420837 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:15.420852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.422628 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.422635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:15.424175 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:15.424189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.425272 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.425280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:15.425534 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:15.426305 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:15.426333 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:15.428313 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:15.431200 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:15.431207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:15.431225 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:15.431650 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:15.432085 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:15.432088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:15.432090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:15.432259 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:15.433005 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:15.433012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:15.433298 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 18:49:15.433489 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 18:49:15.433815 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":119,"threads_started":1}
I20260927 18:49:15.440714 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:15.414426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:45587 (local address 127.29.159.126:41721)
0927 18:49:15.415048 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:15.415080 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:15.415551 (+ 471us) server_negotiation.cc:407] Connection header received
0927 18:49:15.416779 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:15.416789 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:15.419304 (+ 2515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:15.421046 (+ 1742us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:15.421430 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.422478 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.424331 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.425075 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.425253 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:15.428547 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:15.428553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:15.428572 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:15.428604 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:15.431057 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:15.431813 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:15.431820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:15.431825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:15.431973 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:15.432382 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:15.432385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:15.432387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:15.432883 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:15.433021 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:15.433666 (+ 645us) server_negotiation.cc:299] Negotiation successful
0927 18:49:15.433961 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":111,"threads_started":1}
I20260927 18:49:15.443806 10811 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41721
I20260927 18:49:15.444054 10811 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:15.444649 10811 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
W20260927 18:49:15.446956 10811 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:41721 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:49:15.447079 10811 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:15.447253 10811 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
I20260927 18:49:15.448053 10811 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:15.448222 10811 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
W20260927 18:49:15.448935 10811 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:49:15.556203 10815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:15.556536 10815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:15.556653 10815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:15.561916 10815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:15.561996 10815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:15.562083 10815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:49:15.568164 10815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.159.84:58117/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:37469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=43039
--tserver_master_addrs=127.29.159.126:41721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:15.569350 10815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:15.570444 10815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:15.581955 10821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:15.581981 10820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:15.582362 10823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:15.582613 10815 server_base.cc:1034] running on GCE node
I20260927 18:49:15.582989 10815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:15.583714 10815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:15.584904 10815 hybrid_clock.cc:648] HybridClock initialized: now 1790534955584875 us; error 49 us; skew 500 ppm
Sep 27 18:49:15 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:15.588130 10815 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:49:15.589120 10815 webserver.cc:460] Webserver started at http://127.29.159.66:43039/ using document root <none> and password file <none>
I20260927 18:49:15.589710 10815 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:15.589829 10815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:49:15 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:49:15.617022 10815 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260927 18:49:15.620414 10831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:15.622159 10815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:49:15.622352 10815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "47e7a43ef9d94b16916ec65c330e4529"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "d5b3a0d72f8f3f37bcd5b9da645a5021"
server_key_iv: "8201f95fef8217a2d11e7507d82e7640"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:15.623193 10815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:15.658354 10815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:15.662405 10815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:15.662626 10815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:15.663538 10815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:15.664633 10815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:15.664696 10815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:15.664757 10815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:15.664788 10815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:15.675417 10815 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:37469
I20260927 18:49:15.678157 10945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:37469 every 8 connection(s)
I20260927 18:49:15.679543 10815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:15.683413 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 10815
I20260927 18:49:15.683917 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.159.84:58117/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:42741
--local_ip_for_outbound_sockets=127.29.159.67
--tserver_master_addrs=127.29.159.126:41721
--webserver_port=39555
--webserver_interface=127.29.159.67
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.159.84:58117/kms with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:49:15 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:15.701220 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:15.687230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:36859 (local address 127.29.159.126:41721)
0927 18:49:15.687470 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:15.687475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:15.687494 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:49:15.688076 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:15.688080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:15.688135 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:15.688228 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:15.689782 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.690595 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.692372 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.692756 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.692892 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:15.696759 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:15.696763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:15.696766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:15.696804 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:15.698777 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:15.699489 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:15.699493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:15.699495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:15.699588 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:15.699969 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:15.699973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:15.699975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:15.700170 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:15.700295 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:15.700908 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 18:49:15.701048 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260927 18:49:15.702999 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:15.685828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41721 (local address 127.29.159.66:36859)
0927 18:49:15.686445 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:15.686512 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:15.687781 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:15.688530 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:15.688539 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:15.688919 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:15.689608 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:15.689627 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.690761 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.690765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:15.692205 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:15.692220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.692971 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.692979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:15.693218 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:15.694251 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:15.694273 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:15.696569 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:15.698936 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:15.698943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:15.698962 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:15.699338 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:15.699698 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:15.699701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:15.699703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:15.699861 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:15.700296 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:15.700305 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:15.700596 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 18:49:15.701537 (+ 941us) client_negotiation.cc:240] Negotiation successful
0927 18:49:15.701884 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":132,"threads_started":1}
I20260927 18:49:15.704280 10946 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41721
I20260927 18:49:15.704527 10946 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:15.704979 10946 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
W20260927 18:49:15.706521 10946 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:41721 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:49:15.706708 10946 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:15.706905 10946 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
I20260927 18:49:15.707842 10946 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:15.708029 10946 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
W20260927 18:49:15.708791 10946 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:49:15.835147 10948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:15.835521 10948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:15.835585 10948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:15.845350 10948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:15.845496 10948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:15.845661 10948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:49:15.853988 10948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.159.84:58117/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:42741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=39555
--tserver_master_addrs=127.29.159.126:41721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:15.855489 10948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:15.856876 10948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:15.869059 10954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:15.869086 10955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:15.869087 10957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:15.869840 10948 server_base.cc:1034] running on GCE node
I20260927 18:49:15.870335 10948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:15.871054 10948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:15.872211 10948 hybrid_clock.cc:648] HybridClock initialized: now 1790534955872167 us; error 58 us; skew 500 ppm
Sep 27 18:49:15 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:15.875443 10948 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:49:15.876483 10948 webserver.cc:460] Webserver started at http://127.29.159.67:39555/ using document root <none> and password file <none>
I20260927 18:49:15.877254 10948 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:15.877414 10948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:49:15 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:49:15.897876 10948 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:49:15.901160 10965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:15.902848 10948 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:49:15.903014 10948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "c3c2c2020a774358a3751464155fde77"
format_stamp: "Formatted at 2026-09-27 18:49:14 on dist-test-slave-8tg2"
server_key: "c6546fd1962b291dfc3490ff08dc84a5"
server_key_iv: "5dda7009b48b3432b7da3601f44de8a8"
server_key_version: "kuduclusterkey@0"
I20260927 18:49:15.903781 10948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:15.932834 10948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:15.936079 10948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:15.936288 10948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:15.936949 10948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:15.937819 10948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:15.937886 10948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:15.937955 10948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:15.938006 10948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:15.949333 10948 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:42741
I20260927 18:49:15.949361 11078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:42741 every 8 connection(s)
I20260927 18:49:15.950457 10948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:15.957090 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 10948
Sep 27 18:49:15 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:15.967020 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:15.952490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:45885 (local address 127.29.159.126:41721)
0927 18:49:15.952625 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:15.952630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:15.953592 (+ 962us) server_negotiation.cc:407] Connection header received
0927 18:49:15.954639 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:15.954642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:15.954699 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:15.954808 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:15.955981 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.956514 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.958483 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.958797 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.958913 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:15.962975 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:15.962980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:15.962983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:15.963010 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:15.964827 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:15.965345 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:15.965349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:15.965351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:15.965467 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:15.965839 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:15.965843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:15.965845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:15.966068 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:15.966173 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:15.966736 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 18:49:15.966853 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:49:15.967994 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:15.952947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41721 (local address 127.29.159.67:45885)
0927 18:49:15.953436 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:15.953488 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:15.954422 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:15.955001 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:15.955008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:15.955269 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:15.955825 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:15.955840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.956641 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.956644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:15.958358 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:15.958368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.959013 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.959018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:15.959227 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:15.959933 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:15.959954 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:15.962742 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:15.964945 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:15.964950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:15.964964 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:15.965215 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:15.965569 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:15.965572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:15.965574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:15.965731 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:15.966186 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:15.966195 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:15.966491 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 18:49:15.966660 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 18:49:15.966905 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":152,"threads_started":1}
I20260927 18:49:15.969488 11079 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41721
I20260927 18:49:15.969751 11079 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:15.970213 11079 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
I20260927 18:49:15.971221 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:15.957747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.29.159.126:41721)
0927 18:49:15.958613 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:15.958620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:15.958886 (+ 266us) server_negotiation.cc:407] Connection header received
0927 18:49:15.959048 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:15.959053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:15.959094 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:15.959169 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:15.960414 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.961173 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.962423 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:15.962635 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:15.962751 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:15.964985 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:15.964988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:15.964991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:15.965019 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:15.968158 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:15.968795 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:15.968797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:15.968798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:15.968867 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:15.969302 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:15.969306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:15.969308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:15.969528 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:15.969713 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:15.970912 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0927 18:49:15.971029 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":71,"threads_started":1}
I20260927 18:49:15.971622 10613 ts_manager.cc:194] Registered new tserver with Master: c3c2c2020a774358a3751464155fde77 (127.29.159.67:42741)
I20260927 18:49:15.973202 10613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:45885
Sep 27 18:49:16 dist-test-slave-8tg2 krb5kdc[9617](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534956, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:16.449177 10811 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:16.449486 10811 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
I20260927 18:49:16.450238 10613 ts_manager.cc:194] Registered new tserver with Master: abf1a5cb5802426dae1e379464dc78d8 (127.29.159.65:33755)
I20260927 18:49:16.451045 10613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:45587
Sep 27 18:49:16 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534956, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:49:16.709045 10946 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:16.709414 10946 heartbeater.cc:511] Master 127.29.159.126:41721 requested a full tablet report, sending...
I20260927 18:49:16.710721 10613 ts_manager.cc:194] Registered new tserver with Master: 47e7a43ef9d94b16916ec65c330e4529 (127.29.159.66:37469)
I20260927 18:49:16.711526 10613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:36859
I20260927 18:49:16.718262 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:49:16.740922 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:16.720715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.29.159.126:41721)
0927 18:49:16.721047 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:16.721052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:16.722137 (+ 1085us) server_negotiation.cc:407] Connection header received
0927 18:49:16.722195 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:16.722199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:16.722257 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:16.722347 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:16.724343 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.725541 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.727922 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.728271 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.728372 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:16.737318 (+ 8946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:16.737322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:16.737324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:16.737353 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:16.738967 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:16.739459 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:16.739464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:16.739466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:16.739549 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:16.739858 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:16.739861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:16.739863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:16.740051 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:16.740190 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:16.740463 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 18:49:16.740595 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":94,"threads_started":1}
I20260927 18:49:16.744930 10613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:49:16.748483 10613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:49:16.764595 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:16.759168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:42741 (local address 127.0.0.1:46148)
0927 18:49:16.759780 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:16.759796 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:16.759881 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:16.760636 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:16.760642 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:16.760664 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:16.760800 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:16.760807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.761978 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.761984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:16.762858 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:16.762868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.764036 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.764040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:16.764149 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:16.764182 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:49:16.764316 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 18:49:16.764446 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":86,"threads_started":1}
I20260927 18:49:16.764596 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:16.759929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.29.159.67:42741)
0927 18:49:16.760230 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:16.760236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:16.760256 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:49:16.760319 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:16.760323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:16.760490 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:16.760647 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:16.760975 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.761766 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.763002 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.763913 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.764027 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:16.764174 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:16.764358 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 18:49:16.764449 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":108,"threads_started":1}
I20260927 18:49:16.767969 11004 tablet_service.cc:1467] Processing CreateTablet for tablet ceb17119257f415eab36b9f94c23ee1a (DEFAULT_TABLE table=test-table [id=6deb6c3cc0b3457ea85ff8f3a6a25867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:16.769167 11004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb17119257f415eab36b9f94c23ee1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:16.772755 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:16.759539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:37469 (local address 127.0.0.1:38986)
0927 18:49:16.763584 (+ 4045us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:16.763597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:16.763680 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:16.768121 (+ 4441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:16.768125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:16.768144 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:16.768240 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:16.768246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.769888 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.769891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:16.771167 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:16.771179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.772323 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.772327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:16.772442 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:16.772472 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:49:16.772530 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:49:16.772603 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3986,"mutex_wait_us":30,"thread_start_us":85,"threads_started":1}
I20260927 18:49:16.773383 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:16.758830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:33755 (local address 127.0.0.1:46734)
0927 18:49:16.759170 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:16.759214 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:16.759330 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:16.768268 (+ 8938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:16.768271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:16.768289 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:16.768384 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:16.768390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.769930 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.769934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:16.771757 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:16.771768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.772798 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.772802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:16.773107 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:16.773132 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:49:16.773189 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 18:49:16.773251 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260927 18:49:16.773492 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:16.760578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.29.159.66:37469)
0927 18:49:16.767635 (+ 7057us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:16.767642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:16.767668 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:49:16.767745 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:16.767750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:16.767955 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:16.768416 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:16.768564 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.769743 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.771319 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.772178 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.773051 (+ 873us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:16.773197 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:16.773250 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 18:49:16.773344 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6947,"thread_start_us":74,"threads_started":1}
I20260927 18:49:16.774313 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:16.759668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.29.159.65:33755)
0927 18:49:16.767660 (+ 7992us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:16.767667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:16.767691 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:49:16.767764 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:16.767769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:16.767961 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:16.768112 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:16.768611 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.769798 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.771910 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.772659 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.773919 (+ 1260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:16.774050 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:16.774098 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 18:49:16.774183 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7876,"thread_start_us":91,"threads_started":1}
I20260927 18:49:16.776286 10861 tablet_service.cc:1467] Processing CreateTablet for tablet ceb17119257f415eab36b9f94c23ee1a (DEFAULT_TABLE table=test-table [id=6deb6c3cc0b3457ea85ff8f3a6a25867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:16.777228 10861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb17119257f415eab36b9f94c23ee1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:16.786379 10735 tablet_service.cc:1467] Processing CreateTablet for tablet ceb17119257f415eab36b9f94c23ee1a (DEFAULT_TABLE table=test-table [id=6deb6c3cc0b3457ea85ff8f3a6a25867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:16.787617 10735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb17119257f415eab36b9f94c23ee1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:16.790031 11112 tablet_bootstrap.cc:492] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: Bootstrap starting.
I20260927 18:49:16.792454 11112 tablet_bootstrap.cc:654] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:16.793402 11112 log.cc:824] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:16.794248 11113 tablet_bootstrap.cc:492] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: Bootstrap starting.
I20260927 18:49:16.796880 11113 tablet_bootstrap.cc:654] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:16.797595 11113 log.cc:824] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:16.799769 11112 tablet_bootstrap.cc:492] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: No bootstrap required, opened a new log
I20260927 18:49:16.800001 11112 ts_tablet_manager.cc:1397] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260927 18:49:16.800251 11113 tablet_bootstrap.cc:492] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: No bootstrap required, opened a new log
I20260927 18:49:16.800444 11113 ts_tablet_manager.cc:1397] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:49:16.804497 11113 raft_consensus.cc:348] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf1a5cb5802426dae1e379464dc78d8" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33755 } } peers { permanent_uuid: "c3c2c2020a774358a3751464155fde77" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42741 } } peers { permanent_uuid: "47e7a43ef9d94b16916ec65c330e4529" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 37469 } }
I20260927 18:49:16.804497 11112 raft_consensus.cc:348] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf1a5cb5802426dae1e379464dc78d8" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33755 } } peers { permanent_uuid: "c3c2c2020a774358a3751464155fde77" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42741 } } peers { permanent_uuid: "47e7a43ef9d94b16916ec65c330e4529" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 37469 } }
I20260927 18:49:16.804772 11112 raft_consensus.cc:374] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:16.804772 11113 raft_consensus.cc:374] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:16.804823 11112 raft_consensus.cc:729] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c2c2020a774358a3751464155fde77, State: Initialized, Role: FOLLOWER
I20260927 18:49:16.804832 11113 raft_consensus.cc:729] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e7a43ef9d94b16916ec65c330e4529, State: Initialized, Role: FOLLOWER
I20260927 18:49:16.805315 11112 consensus_queue.cc:260] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf1a5cb5802426dae1e379464dc78d8" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33755 } } peers { permanent_uuid: "c3c2c2020a774358a3751464155fde77" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42741 } } peers { permanent_uuid: "47e7a43ef9d94b16916ec65c330e4529" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 37469 } }
I20260927 18:49:16.805646 11113 consensus_queue.cc:260] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf1a5cb5802426dae1e379464dc78d8" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33755 } } peers { permanent_uuid: "c3c2c2020a774358a3751464155fde77" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42741 } } peers { permanent_uuid: "47e7a43ef9d94b16916ec65c330e4529" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 37469 } }
I20260927 18:49:16.806420 11112 ts_tablet_manager.cc:1428] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:49:16.806710 11113 ts_tablet_manager.cc:1428] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:49:16.806790 11079 heartbeater.cc:503] Master 127.29.159.126:41721 was elected leader, sending a full tablet report...
I20260927 18:49:16.807161 10946 heartbeater.cc:503] Master 127.29.159.126:41721 was elected leader, sending a full tablet report...
I20260927 18:49:16.807829 11115 tablet_bootstrap.cc:492] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8: Bootstrap starting.
I20260927 18:49:16.810333 11115 tablet_bootstrap.cc:654] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:16.811236 11115 log.cc:824] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:16.819976 11115 tablet_bootstrap.cc:492] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8: No bootstrap required, opened a new log
I20260927 18:49:16.820204 11115 ts_tablet_manager.cc:1397] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260927 18:49:16.824429 11115 raft_consensus.cc:348] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf1a5cb5802426dae1e379464dc78d8" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33755 } } peers { permanent_uuid: "c3c2c2020a774358a3751464155fde77" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42741 } } peers { permanent_uuid: "47e7a43ef9d94b16916ec65c330e4529" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 37469 } }
I20260927 18:49:16.824723 11115 raft_consensus.cc:374] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:16.824774 11115 raft_consensus.cc:729] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf1a5cb5802426dae1e379464dc78d8, State: Initialized, Role: FOLLOWER
I20260927 18:49:16.825263 11115 consensus_queue.cc:260] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf1a5cb5802426dae1e379464dc78d8" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33755 } } peers { permanent_uuid: "c3c2c2020a774358a3751464155fde77" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42741 } } peers { permanent_uuid: "47e7a43ef9d94b16916ec65c330e4529" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 37469 } }
I20260927 18:49:16.825973 10811 heartbeater.cc:503] Master 127.29.159.126:41721 was elected leader, sending a full tablet report...
I20260927 18:49:16.826397 11115 ts_tablet_manager.cc:1428] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 18:49:16.913959 10812 tablet.cc:2369] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:49:16.938982 10947 tablet.cc:2369] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:49:16.952489 11080 tablet.cc:2369] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:49:16 dist-test-slave-8tg2 krb5kdc[9617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534956, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for HTTP/127.29.159.84@KRBTEST.COM
I20260927 18:49:16.994767 11117 raft_consensus.cc:482] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:16.995050 11117 raft_consensus.cc:504] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf1a5cb5802426dae1e379464dc78d8" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33755 } } peers { permanent_uuid: "c3c2c2020a774358a3751464155fde77" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42741 } } peers { permanent_uuid: "47e7a43ef9d94b16916ec65c330e4529" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 37469 } }
I20260927 18:49:16.996302 11117 leader_election.cc:288] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abf1a5cb5802426dae1e379464dc78d8 (127.29.159.65:33755), 47e7a43ef9d94b16916ec65c330e4529 (127.29.159.66:37469)
I20260927 18:49:17.004271 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:16.996956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:33445 (local address 127.29.159.65:33755)
0927 18:49:16.997097 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:16.997102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:16.997328 (+ 226us) server_negotiation.cc:407] Connection header received
0927 18:49:16.997776 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:16.997779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:16.997829 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:16.997878 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:16.998534 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.999331 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:17.001986 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:17.002591 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:17.002687 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:17.002778 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:17.004028 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0927 18:49:17.004125 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:49:17.004709 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:16.996463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:33755 (local address 127.29.159.67:33445)
0927 18:49:16.997235 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:16.997250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:16.997627 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:16.998025 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:16.998028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:16.998048 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:16.998402 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:16.998414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:16.999720 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:16.999723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:17.001847 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:17.001858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:17.003700 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:17.003704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:17.003851 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:17.003878 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:49:17.004459 (+ 581us) client_negotiation.cc:240] Negotiation successful
0927 18:49:17.004547 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":473,"threads_started":1}
I20260927 18:49:17.005496 10754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb17119257f415eab36b9f94c23ee1a" candidate_uuid: "c3c2c2020a774358a3751464155fde77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abf1a5cb5802426dae1e379464dc78d8" is_pre_election: true
I20260927 18:49:17.005854 10754 raft_consensus.cc:2454] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3c2c2020a774358a3751464155fde77 in term 0.
I20260927 18:49:17.006417 10969 leader_election.cc:302] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abf1a5cb5802426dae1e379464dc78d8, c3c2c2020a774358a3751464155fde77; no voters:
I20260927 18:49:17.006717 11117 raft_consensus.cc:2781] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:49:17.006809 11117 raft_consensus.cc:482] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:49:17.006855 11117 raft_consensus.cc:3037] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:17.008215 11117 raft_consensus.cc:504] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf1a5cb5802426dae1e379464dc78d8" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33755 } } peers { permanent_uuid: "c3c2c2020a774358a3751464155fde77" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42741 } } peers { permanent_uuid: "47e7a43ef9d94b16916ec65c330e4529" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 37469 } }
I20260927 18:49:17.008652 11117 leader_election.cc:288] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [CANDIDATE]: Term 1 election: Requested vote from peers abf1a5cb5802426dae1e379464dc78d8 (127.29.159.65:33755), 47e7a43ef9d94b16916ec65c330e4529 (127.29.159.66:37469)
I20260927 18:49:17.009050 10754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb17119257f415eab36b9f94c23ee1a" candidate_uuid: "c3c2c2020a774358a3751464155fde77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abf1a5cb5802426dae1e379464dc78d8"
I20260927 18:49:17.009183 10754 raft_consensus.cc:3037] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:17.010665 10754 raft_consensus.cc:2454] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3c2c2020a774358a3751464155fde77 in term 1.
I20260927 18:49:17.011137 10969 leader_election.cc:302] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abf1a5cb5802426dae1e379464dc78d8, c3c2c2020a774358a3751464155fde77; no voters:
I20260927 18:49:17.011413 11117 raft_consensus.cc:2781] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:17.011763 11117 raft_consensus.cc:686] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 1 LEADER]: Becoming Leader. State: Replica: c3c2c2020a774358a3751464155fde77, State: Running, Role: LEADER
I20260927 18:49:17.012204 11117 consensus_queue.cc:237] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf1a5cb5802426dae1e379464dc78d8" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33755 } } peers { permanent_uuid: "c3c2c2020a774358a3751464155fde77" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42741 } } peers { permanent_uuid: "47e7a43ef9d94b16916ec65c330e4529" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 37469 } }
I20260927 18:49:17.015625 10612 catalog_manager.cc:5495] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3c2c2020a774358a3751464155fde77 (127.29.159.67). New cstate: current_term: 1 leader_uuid: "c3c2c2020a774358a3751464155fde77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf1a5cb5802426dae1e379464dc78d8" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3c2c2020a774358a3751464155fde77" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47e7a43ef9d94b16916ec65c330e4529" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 37469 } health_report { overall_health: UNKNOWN } } }
I20260927 18:49:17.023141 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:16.996975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:50961 (local address 127.29.159.66:37469)
0927 18:49:16.997199 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:16.997203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:16.997567 (+ 364us) server_negotiation.cc:407] Connection header received
0927 18:49:16.998400 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:16.998404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:16.998462 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:16.998514 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:16.999441 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:17.000180 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:17.002809 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:17.003380 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:17.009897 (+ 6517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:17.009962 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:17.022852 (+ 12890us) server_negotiation.cc:299] Negotiation successful
0927 18:49:17.022978 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260927 18:49:17.023804 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:16.996648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:37469 (local address 127.29.159.67:50961)
0927 18:49:16.997479 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:16.997493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:16.998241 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:16.999150 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:16.999153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:16.999172 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:16.999274 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:16.999279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:17.000479 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:17.000482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:17.002672 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:17.002681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:17.022510 (+ 19829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:17.022516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:17.022647 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:17.022679 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:49:17.023531 (+ 852us) client_negotiation.cc:240] Negotiation successful
0927 18:49:17.023647 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":424,"threads_started":1}
I20260927 18:49:17.024735 10900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb17119257f415eab36b9f94c23ee1a" candidate_uuid: "c3c2c2020a774358a3751464155fde77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e7a43ef9d94b16916ec65c330e4529" is_pre_election: true
I20260927 18:49:17.025074 10900 raft_consensus.cc:2454] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3c2c2020a774358a3751464155fde77 in term 0.
I20260927 18:49:17.025424 10899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb17119257f415eab36b9f94c23ee1a" candidate_uuid: "c3c2c2020a774358a3751464155fde77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e7a43ef9d94b16916ec65c330e4529"
I20260927 18:49:17.025532 10899 raft_consensus.cc:3037] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:17.026937 10899 raft_consensus.cc:2454] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3c2c2020a774358a3751464155fde77 in term 1.
I20260927 18:49:17.063835 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:17.055187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.29.159.67:42741)
0927 18:49:17.055326 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:17.055331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:17.055347 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:49:17.055481 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:17.055484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:17.055564 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:17.055681 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:49:17.059080 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:17.059937 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:17.061411 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:17.061763 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:17.061888 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:17.062384 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:49:17.063009 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:49:17.063138 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:17.063562 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 18:49:17.063647 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:49:17.073675 10754 raft_consensus.cc:1261] T ceb17119257f415eab36b9f94c23ee1a P abf1a5cb5802426dae1e379464dc78d8 [term 1 FOLLOWER]: Refusing update from remote peer c3c2c2020a774358a3751464155fde77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:17.073676 10899 raft_consensus.cc:1261] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529 [term 1 FOLLOWER]: Refusing update from remote peer c3c2c2020a774358a3751464155fde77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:17.075274 11117 consensus_queue.cc:1035] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e7a43ef9d94b16916ec65c330e4529" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 37469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:17.075894 11117 consensus_queue.cc:1035] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abf1a5cb5802426dae1e379464dc78d8" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:17.090102 11128 mvcc.cc:205] Tried to move back new op lower bound from 7334031184162521088 to 7334031183925506048. Current Snapshot: MvccSnapshot[applied={T|T < 7334031184162521088}]
I20260927 18:49:17.092411 11129 mvcc.cc:205] Tried to move back new op lower bound from 7334031184162521088 to 7334031183925506048. Current Snapshot: MvccSnapshot[applied={T|T < 7334031184162521088}]
I20260927 18:49:17.105554 10613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:17.105808 10613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:17.108824 10613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da8b270621eb438091ce19ec34e04ed2: Sending DeleteTablet for 3 replicas of tablet ceb17119257f415eab36b9f94c23ee1a
I20260927 18:49:17.109637 10735 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb17119257f415eab36b9f94c23ee1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:17 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:46734
I20260927 18:49:17.110596 11004 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb17119257f415eab36b9f94c23ee1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:17 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:46148
I20260927 18:49:17.111474 10861 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb17119257f415eab36b9f94c23ee1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:17 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:38986
I20260927 18:49:17.113477 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 10651
I20260927 18:49:17.123704 11139 tablet_replica.cc:330] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: stopping tablet replica
I20260927 18:49:17.124177 11139 raft_consensus.cc:2229] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:17.124660 11139 raft_consensus.cc:2258] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:17.124776 11138 tablet_replica.cc:330] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: stopping tablet replica
I20260927 18:49:17.125583 11138 raft_consensus.cc:2229] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:49:17.126049 11138 raft_consensus.cc:2258] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:17.126654 11139 ts_tablet_manager.cc:1905] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 18:49:17.126711 10600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:33755 (error 108)
I20260927 18:49:17.128271 11138 ts_tablet_manager.cc:1905] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:17.131270 11138 ts_tablet_manager.cc:1918] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:17.131386 11138 log.cc:1196] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/ceb17119257f415eab36b9f94c23ee1a
I20260927 18:49:17.131788 11138 ts_tablet_manager.cc:1939] T ceb17119257f415eab36b9f94c23ee1a P c3c2c2020a774358a3751464155fde77: Deleting consensus metadata
I20260927 18:49:17.131954 11139 ts_tablet_manager.cc:1918] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:17.132076 11139 log.cc:1196] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/ceb17119257f415eab36b9f94c23ee1a
I20260927 18:49:17.132522 11139 ts_tablet_manager.cc:1939] T ceb17119257f415eab36b9f94c23ee1a P 47e7a43ef9d94b16916ec65c330e4529: Deleting consensus metadata
I20260927 18:49:17.133180 10598 catalog_manager.cc:4841] TS c3c2c2020a774358a3751464155fde77 (127.29.159.67:42741): tablet ceb17119257f415eab36b9f94c23ee1a (table test-table [id=6deb6c3cc0b3457ea85ff8f3a6a25867]) successfully deleted
I20260927 18:49:17.133895 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 10815
I20260927 18:49:17.133898 10598 catalog_manager.cc:4841] TS 47e7a43ef9d94b16916ec65c330e4529 (127.29.159.66:37469): tablet ceb17119257f415eab36b9f94c23ee1a (table test-table [id=6deb6c3cc0b3457ea85ff8f3a6a25867]) successfully deleted
I20260927 18:49:17.135901 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:17.135548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:33755 (local address 127.0.0.1:46746)
0927 18:49:17.135732 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:17.135814 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 18:49:17.136090 10600 catalog_manager.cc:4584] TS abf1a5cb5802426dae1e379464dc78d8 (127.29.159.65:33755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ceb17119257f415eab36b9f94c23ee1a: Network error: Client connection negotiation failed: client connection to 127.29.159.65:33755: connect: Connection refused (error 111)
I20260927 18:49:17.159653 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 10948
I20260927 18:49:17.169723 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 10580
I20260927 18:49:17.178138 30333 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:49:17.522832 30333 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:49:17.522951 30333 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:49:17.868048 30333 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:49:17.868 UTC [9647] LOG: received smart shutdown request
2026-09-27 18:49:17.871 UTC [9647] LOG: background worker "logical replication launcher" (PID 9656) exited with exit code 1
2026-09-27 18:49:17.872 UTC [9651] LOG: shutting down
2026-09-27 18:49:18.051 UTC [9647] LOG: database system is shut down
2026-09-27T18:49:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49031 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:49:18 dist-test-slave-8tg2 krb5kdc[11149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:49:18 dist-test-slave-8tg2 krb5kdc[11149](info): set up 2 sockets
Sep 27 18:49:18 dist-test-slave-8tg2 krb5kdc[11149](info): commencing operation
krb5kdc: starting...
W20260927 18:49:19.215557 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:49:19.343788 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:45525
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:45525,127.29.159.125:39833,127.29.159.124:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:19.453316 11162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:19.453637 11162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:19.453727 11162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:19.457795 11162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:19.457898 11162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:19.457937 11162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:19.457974 11162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:19.458007 11162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:19.463366 11162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:45525,127.29.159.125:39833,127.29.159.124:34749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:19.464565 11162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:19.465648 11162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:19.474201 11167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:19.474201 11168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:19.474407 11170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:19.474545 11162 server_base.cc:1034] running on GCE node
I20260927 18:49:19.475127 11162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:19.476284 11162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:19.477475 11162 hybrid_clock.cc:648] HybridClock initialized: now 1790534959477448 us; error 41 us; skew 500 ppm
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:19.480330 11162 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:49:19.481244 11162 webserver.cc:460] Webserver started at http://127.29.159.126:44601/ using document root <none> and password file <none>
I20260927 18:49:19.481813 11162 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:19.481909 11162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:19.482146 11162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:19.484289 11162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "6b33a58cdeec408090356e91b24131d8"
format_stamp: "Formatted at 2026-09-27 18:49:19 on dist-test-slave-8tg2"
server_key: "8e6e4a97273337b99dadcde038c62767"
server_key_iv: "557c9e8782654f737952637eca1940be"
server_key_version: "encryptionkey@0"
I20260927 18:49:19.484860 11162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "6b33a58cdeec408090356e91b24131d8"
format_stamp: "Formatted at 2026-09-27 18:49:19 on dist-test-slave-8tg2"
server_key: "8e6e4a97273337b99dadcde038c62767"
server_key_iv: "557c9e8782654f737952637eca1940be"
server_key_version: "encryptionkey@0"
I20260927 18:49:19.488698 11162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:49:19.491050 11177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:19.492341 11162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:49:19.492494 11162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "6b33a58cdeec408090356e91b24131d8"
format_stamp: "Formatted at 2026-09-27 18:49:19 on dist-test-slave-8tg2"
server_key: "8e6e4a97273337b99dadcde038c62767"
server_key_iv: "557c9e8782654f737952637eca1940be"
server_key_version: "encryptionkey@0"
I20260927 18:49:19.492604 11162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:19.518496 11162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:19.522383 11162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:19.522601 11162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:19.529944 11162 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:45525
I20260927 18:49:19.530224 11229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:45525 every 8 connection(s)
I20260927 18:49:19.531011 11162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:19.533875 11230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:19.538054 11230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:49:19.539649 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 11162
I20260927 18:49:19.539800 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:49:19.540074 30333 external_mini_cluster.cc:1428] Setting key a44460bd0d191d93b787e7ca12ec0d4d
WARNING: no policy specified for kudu/127.29.159.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.125@KRBTEST.COM" created.
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:19.557682 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.540557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.29.159.126:45525)
0927 18:49:19.541131 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.541144 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.541193 (+ 49us) server_negotiation.cc:407] Connection header received
0927 18:49:19.542528 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.542534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.542715 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.542872 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.544039 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.545020 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.546663 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.547022 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.547138 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.552612 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:19.552618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:19.552630 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:19.552653 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:19.554342 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.554911 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.554916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.554917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.555018 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.555382 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.555385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.555386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.555923 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:19.556033 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:19.556515 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 18:49:19.556837 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":447,"thread_start_us":160,"threads_started":1}
I20260927 18:49:19.557684 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.540555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:45525 (local address 127.0.0.1:53968)
0927 18:49:19.541035 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.541095 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:19.542186 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:19.542925 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:19.542934 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:19.543459 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:19.543849 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.543870 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.545160 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.545166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.546537 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.546552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.547196 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.547203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.547356 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.548021 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:19.548046 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:19.552460 (+ 4414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:19.554443 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.554455 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.554469 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.554792 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.555116 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.555122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.555127 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.555283 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.556047 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:19.556056 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:19.556237 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 18:49:19.556458 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 18:49:19.556921 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":82,"thread_start_us":148,"threads_started":1}
I20260927 18:49:19.560559 11230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:49:19.561365 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.561036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:39833 (local address 127.0.0.1:37878)
0927 18:49:19.561233 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.561307 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:39833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260927 18:49:19.561573 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.159.125:39833: connect: Connection refused (error 111)
I20260927 18:49:19.562455 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.562256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:39833 (local address 127.0.0.1:37892)
0927 18:49:19.562362 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.562406 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:39833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 18:49:19.562597 11230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.125:39833: Network error: Client connection negotiation failed: client connection to 127.29.159.125:39833: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab.
Entry for principal kudu/127.29.159.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab.
I20260927 18:49:19.569957 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.125:39833
--webserver_interface=127.29.159.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab
--principal=kudu/127.29.159.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:45525,127.29.159.125:39833,127.29.159.124:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:19.611876 11230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } attempt: 1
I20260927 18:49:19.612784 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.612476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:39833 (local address 127.0.0.1:37898)
0927 18:49:19.612629 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.612702 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:39833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 18:49:19.613575 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.613336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:39833 (local address 127.0.0.1:37900)
0927 18:49:19.613466 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.613517 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:39833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 18:49:19.613758 11230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.125:39833: Network error: Client connection negotiation failed: client connection to 127.29.159.125:39833: connect: Connection refused (error 111)
W20260927 18:49:19.681589 11237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:19.681864 11237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:19.681907 11237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 18:49:19.681975 11230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } attempt: 2
I20260927 18:49:19.682897 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.682550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:39833 (local address 127.0.0.1:37902)
0927 18:49:19.682749 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.682818 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:39833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 18:49:19.683775 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.683527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:39833 (local address 127.0.0.1:37918)
0927 18:49:19.683659 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.683707 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:39833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 18:49:19.683934 11230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.125:39833: Network error: Client connection negotiation failed: client connection to 127.29.159.125:39833: connect: Connection refused (error 111)
W20260927 18:49:19.686666 11237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:19.686717 11237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:19.686738 11237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:19.686754 11237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:19.686770 11237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:19.692185 11237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:45525,127.29.159.125:39833,127.29.159.124:34749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab
--principal=kudu/127.29.159.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.125:39833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.159.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:19.693202 11237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:19.694169 11237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:19.702790 11242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:19.702790 11245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:19.702865 11237 server_base.cc:1034] running on GCE node
W20260927 18:49:19.703194 11243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:19.703797 11237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:19.704881 11237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:19.706039 11237 hybrid_clock.cc:648] HybridClock initialized: now 1790534959706024 us; error 29 us; skew 500 ppm
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:19.708806 11237 init.cc:377] Logged in from keytab as kudu/127.29.159.125@KRBTEST.COM (short username kudu)
I20260927 18:49:19.709609 11237 webserver.cc:460] Webserver started at http://127.29.159.125:39173/ using document root <none> and password file <none>
I20260927 18:49:19.710111 11237 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:19.710167 11237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:19.710325 11237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:19.712340 11237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/data/instance:
uuid: "43acae53a4014ef38ad753604c3ab162"
format_stamp: "Formatted at 2026-09-27 18:49:19 on dist-test-slave-8tg2"
server_key: "c301c8af42fd27ed2689e99cb4df7a4a"
server_key_iv: "dc90590b06b34b0ce3dbb88033097906"
server_key_version: "encryptionkey@0"
I20260927 18:49:19.712834 11237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/wal/instance:
uuid: "43acae53a4014ef38ad753604c3ab162"
format_stamp: "Formatted at 2026-09-27 18:49:19 on dist-test-slave-8tg2"
server_key: "c301c8af42fd27ed2689e99cb4df7a4a"
server_key_iv: "dc90590b06b34b0ce3dbb88033097906"
server_key_version: "encryptionkey@0"
I20260927 18:49:19.716480 11237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:49:19.718863 11252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:19.720091 11237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:49:19.720211 11237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/wal
uuid: "43acae53a4014ef38ad753604c3ab162"
format_stamp: "Formatted at 2026-09-27 18:49:19 on dist-test-slave-8tg2"
server_key: "c301c8af42fd27ed2689e99cb4df7a4a"
server_key_iv: "dc90590b06b34b0ce3dbb88033097906"
server_key_version: "encryptionkey@0"
I20260927 18:49:19.720312 11237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:19.729532 11237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:19.732328 11237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:19.732519 11237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:19.739249 11237 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.125:39833
I20260927 18:49:19.739586 11304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.125:39833 every 8 connection(s)
I20260927 18:49:19.740222 11237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
I20260927 18:49:19.742928 11305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:19.746186 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 11237
I20260927 18:49:19.746291 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-1/wal/instance
I20260927 18:49:19.746563 30333 external_mini_cluster.cc:1428] Setting key e92be28568d70dc70ca3c3b69ef55060
I20260927 18:49:19.746801 11305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.159.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.124@KRBTEST.COM" created.
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:19.763295 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.748703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53978 (local address 127.29.159.126:45525)
0927 18:49:19.748864 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.748869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.749758 (+ 889us) server_negotiation.cc:407] Connection header received
0927 18:49:19.751073 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.751075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.751123 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.751199 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.752896 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.753485 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.755303 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.755621 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.755716 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.759133 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:19.759136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:19.759138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:19.759166 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:19.760921 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.761507 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.761513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.761518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.761614 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.762025 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.762030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.762034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.762281 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:19.762402 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:19.762906 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 18:49:19.763060 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:49:19.764037 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.749084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:45525 (local address 127.0.0.1:53978)
0927 18:49:19.749579 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.749635 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:19.750807 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:19.751418 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:19.751448 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:19.751988 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:19.752703 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.752727 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.753625 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.753630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.755172 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.755181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.755762 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.755765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.755988 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.756578 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:19.756594 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:19.758946 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:19.761061 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.761066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.761078 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.761389 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.761712 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.761715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.761717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.761920 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.762396 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:19.762401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:19.762641 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 18:49:19.762805 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 18:49:19.763055 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":105,"threads_started":1}
I20260927 18:49:19.765581 11305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab.
Entry for principal kudu/127.29.159.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab.
I20260927 18:49:19.775089 11230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } attempt: 3
I20260927 18:49:19.775226 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.766145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:39833 (local address 127.0.0.1:37934)
0927 18:49:19.766272 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.766287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:19.766358 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:19.766934 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:19.766936 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:19.767062 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:19.767149 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.767153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.768329 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.768334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.768977 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.768986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.769623 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.769627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.769706 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.770181 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:19.770219 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:19.771582 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:19.773579 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.773581 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.773583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.773774 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.774078 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.774081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.774083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.774211 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.774725 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:19.774728 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:19.774809 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 18:49:19.774938 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 18:49:19.775063 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 18:49:19.775274 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.124:34749
--webserver_interface=127.29.159.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab
--principal=kudu/127.29.159.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:45525,127.29.159.125:39833,127.29.159.124:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:19.776031 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.766234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.29.159.125:39833)
0927 18:49:19.766551 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.766568 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.766588 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:49:19.766623 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.766627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.766812 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.766943 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.767321 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.768196 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.769085 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.769490 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.769578 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.771707 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:19.771715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:19.771727 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:19.771748 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:19.773486 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.773891 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.773896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.773897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.773972 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.774315 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.774320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.774322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.774621 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:19.774764 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:19.775019 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 18:49:19.775831 (+ 812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":143,"threads_started":1}
I20260927 18:49:19.778192 11305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:49:19.778935 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.778667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38242)
0927 18:49:19.778790 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.778865 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:49:19.779194 11256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
I20260927 18:49:19.780397 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.780165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38244)
0927 18:49:19.780281 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.780330 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
W20260927 18:49:19.780826 11305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:34749: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
I20260927 18:49:19.784204 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.775758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.29.159.125:39833)
0927 18:49:19.776208 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.776212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.776224 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:49:19.776262 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.776264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.776297 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.776356 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.776923 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.777560 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.778608 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.778954 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.779118 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.781195 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:19.781199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:19.781201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:19.781223 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:19.782499 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.782948 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.782952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.782954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.783041 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.783330 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.783333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.783335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.783604 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:19.783710 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:19.783963 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 18:49:19.784063 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":117,"threads_started":1}
I20260927 18:49:19.784204 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.775527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:39833 (local address 127.0.0.1:37948)
0927 18:49:19.775680 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.775699 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:19.775785 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:19.776409 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:19.776416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:19.776615 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:19.776763 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.776772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.777676 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.777682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.778487 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.778499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.779121 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.779125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.779251 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.779725 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:19.779736 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:19.781063 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:19.782628 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.782631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.782633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.782831 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.783134 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.783140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.783142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.783233 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.783701 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:19.783704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:19.783807 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 18:49:19.783914 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:49:19.784048 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 18:49:19.785215 11230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:49:19.785900 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.785692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38248)
0927 18:49:19.785802 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.785846 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 18:49:19.786918 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.786698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38260)
0927 18:49:19.786831 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.786872 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260927 18:49:19.787118 11230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:34749: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
I20260927 18:49:19.818298 11230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } attempt: 1
I20260927 18:49:19.819216 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.818908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38266)
0927 18:49:19.819072 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.819138 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 18:49:19.820045 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.819786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38268)
0927 18:49:19.819901 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.819955 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 18:49:19.820186 11230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:34749: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
I20260927 18:49:19.830116 11305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } attempt: 1
I20260927 18:49:19.830925 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.830647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38272)
0927 18:49:19.830791 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.830850 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 18:49:19.831631 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.831406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38276)
0927 18:49:19.831528 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.831579 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 18:49:19.831759 11305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:34749: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
W20260927 18:49:19.891474 11312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:19.891878 11312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:19.891956 11312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 18:49:19.895339 11230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } attempt: 2
W20260927 18:49:19.896101 11312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:19.896178 11312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:19.896221 11312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260927 18:49:19.896227 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.895931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38278)
0927 18:49:19.896053 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.896117 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 18:49:19.896256 11312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:19.896281 11312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:19.896919 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.896696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38286)
0927 18:49:19.896808 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.896854 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 18:49:19.897074 11230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:34749: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
I20260927 18:49:19.899911 11305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } attempt: 2
I20260927 18:49:19.900560 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.900308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38298)
0927 18:49:19.900438 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.900489 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 18:49:19.901289 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.901107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38314)
0927 18:49:19.901202 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.901243 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 18:49:19.901443 11305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:34749: Network error: Client connection negotiation failed: client connection to 127.29.159.124:34749: connect: Connection refused (error 111)
I20260927 18:49:19.902380 11312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:45525,127.29.159.125:39833,127.29.159.124:34749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab
--principal=kudu/127.29.159.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.124:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.159.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:19.903589 11312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:19.904656 11312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:49:19.913419 11312 server_base.cc:1034] running on GCE node
W20260927 18:49:19.913381 11319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:19.913381 11318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:19.913389 11321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:19.914211 11312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:19.915316 11312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:19.916530 11312 hybrid_clock.cc:648] HybridClock initialized: now 1790534959916514 us; error 32 us; skew 500 ppm
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:19.919363 11312 init.cc:377] Logged in from keytab as kudu/127.29.159.124@KRBTEST.COM (short username kudu)
I20260927 18:49:19.920219 11312 webserver.cc:460] Webserver started at http://127.29.159.124:45635/ using document root <none> and password file <none>
I20260927 18:49:19.920729 11312 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:19.920785 11312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:19.920943 11312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:19.922889 11312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/data/instance:
uuid: "c3a71b71ab5a4fedbd042ed49d53cb61"
format_stamp: "Formatted at 2026-09-27 18:49:19 on dist-test-slave-8tg2"
server_key: "42b0328fefd76fede104c3981b37c4fe"
server_key_iv: "40a828274b52d858fd13313a1348b397"
server_key_version: "encryptionkey@0"
I20260927 18:49:19.923383 11312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/wal/instance:
uuid: "c3a71b71ab5a4fedbd042ed49d53cb61"
format_stamp: "Formatted at 2026-09-27 18:49:19 on dist-test-slave-8tg2"
server_key: "42b0328fefd76fede104c3981b37c4fe"
server_key_iv: "40a828274b52d858fd13313a1348b397"
server_key_version: "encryptionkey@0"
I20260927 18:49:19.927000 11312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:49:19.929430 11328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:19.930608 11312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 18:49:19.930701 11312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/wal
uuid: "c3a71b71ab5a4fedbd042ed49d53cb61"
format_stamp: "Formatted at 2026-09-27 18:49:19 on dist-test-slave-8tg2"
server_key: "42b0328fefd76fede104c3981b37c4fe"
server_key_iv: "40a828274b52d858fd13313a1348b397"
server_key_version: "encryptionkey@0"
I20260927 18:49:19.930774 11312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:19.949218 11312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:19.952363 11312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:19.952528 11312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:19.960175 11312 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.124:34749
I20260927 18:49:19.960503 11380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.124:34749 every 8 connection(s)
I20260927 18:49:19.961227 11312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
I20260927 18:49:19.962419 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 11312
I20260927 18:49:19.962543 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534801142164-30333-0/minicluster-data/master-2/wal/instance
I20260927 18:49:19.962867 30333 external_mini_cluster.cc:1428] Setting key 689a18a5c5fd45c7cb2ee9b2311deed4
I20260927 18:49:19.964740 11381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:19.968518 11381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:19.977545 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.964245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.29.159.124:34749)
0927 18:49:19.964678 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.964695 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.964731 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:49:19.965422 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.965439 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.965761 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.966138 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.966924 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.967982 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.968827 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.969351 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.969516 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.972755 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:19.972761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:19.972769 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:19.972799 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:19.974587 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.975074 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.975079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.975083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.975196 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.975602 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.975607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.975608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.976039 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:19.976257 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:19.976570 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 18:49:19.976830 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":144,"threads_started":1}
I20260927 18:49:19.979655 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.970195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.29.159.126:45525)
0927 18:49:19.970339 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.970344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.970655 (+ 311us) server_negotiation.cc:407] Connection header received
0927 18:49:19.970779 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.970784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.970847 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.970954 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.971637 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.972399 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.973534 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.973927 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.974054 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.976286 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:19.976289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:19.976291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:19.976315 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:19.977856 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.978318 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.978323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.978325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.978403 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.978689 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.978692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.978694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.978870 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:19.978976 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:19.979367 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 18:49:19.979499 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:49:19.979665 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.970150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:45525 (local address 127.0.0.1:53990)
0927 18:49:19.970510 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.970557 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:19.970675 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:19.971001 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:19.971005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:19.971325 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:19.971507 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.971514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.972533 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.972537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.973407 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.973417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.974051 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.974053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.974137 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.974617 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:19.974629 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:19.976136 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:19.977963 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.977974 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.977979 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.978208 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.978501 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.978504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.978506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.978588 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.978975 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:19.978982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:19.979092 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 18:49:19.979290 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 18:49:19.979489 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":134,"threads_started":1}
I20260927 18:49:19.981070 11381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
I20260927 18:49:19.989012 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.979734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.29.159.126:45525)
0927 18:49:19.980091 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.980100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.980112 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:49:19.980144 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.980147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.980186 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.980279 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.981041 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.981758 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.982780 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.983137 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.983253 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.985682 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:19.985688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:19.985693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:19.985735 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:19.987271 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.987787 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.987792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.987794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.987874 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.988151 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.988156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.988157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.988344 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:19.988566 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:19.988692 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 18:49:19.988818 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":107,"threads_started":1}
I20260927 18:49:19.990123 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.981700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.29.159.125:39833)
0927 18:49:19.981850 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.981857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.981876 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:49:19.982024 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.982026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.982065 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.982126 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.982598 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.983304 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.984471 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.984679 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.984834 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.986959 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:19.986963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:19.986965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:19.986990 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:19.988408 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.988825 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.988830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.988831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.988923 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.989217 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.989222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.989224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.989452 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:19.989551 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:19.989902 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 18:49:19.989991 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:49:19.990126 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.981571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:39833 (local address 127.0.0.1:37958)
0927 18:49:19.981806 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.981839 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:19.981909 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:19.982168 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:19.982171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:19.982386 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:19.982488 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.982494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.983415 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.983446 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.984323 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.984336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.984836 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.984839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.984929 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.985353 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:19.985368 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:19.986818 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:19.988520 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.988524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.988527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.988719 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.989033 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.989036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.989037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.989138 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.989571 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:19.989577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:19.989693 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 18:49:19.989841 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 18:49:19.989974 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260927 18:49:19.990918 11381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:49:19.992641 11305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } attempt: 3
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
I20260927 18:49:19.996232 11230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } attempt: 3
I20260927 18:49:19.999724 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.991468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.29.159.124:34749)
0927 18:49:19.991644 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.991650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.991663 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:49:19.991692 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.991695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.991725 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.991815 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.992162 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.992730 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.993749 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.993983 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.994088 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.995989 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:19.995996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:19.995999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:19.996024 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:19.997656 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.998417 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.998420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.998422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.998502 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.998822 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.998825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.998827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.999013 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:19.999089 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:19.999405 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 18:49:19.999543 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
Sep 27 18:49:19 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
I20260927 18:49:20.000348 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.991376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38334)
0927 18:49:19.991540 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.991552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:19.991618 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:19.991825 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:19.991827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:19.991982 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:19.992048 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.992054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.992843 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.992846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.993620 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.993634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.994087 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.994090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.994161 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.994551 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:19.994561 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:19.995862 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:19.997763 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.997769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.997771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.998310 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.998603 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:19.998608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:19.998613 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:19.998719 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:19.999152 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:19.999155 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:19.999241 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 18:49:20.000059 (+ 818us) client_negotiation.cc:240] Negotiation successful
0927 18:49:20.000186 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 18:49:20.001359 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.990926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37966 (local address 127.29.159.125:39833)
0927 18:49:19.991017 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.991022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.991062 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:49:19.991204 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.991207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.991253 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.991346 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.991971 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.992524 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.993598 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.993919 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.994019 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.997064 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:19.997068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:19.997074 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:19.997098 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:19.999368 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:19.999860 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:19.999865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:19.999869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:19.999960 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:20.000281 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:20.000285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:20.000286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:20.000507 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:20.000726 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:20.001100 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 18:49:20.001217 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 18:49:20 dist-test-slave-8tg2 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
I20260927 18:49:20.003948 11387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.993170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.29.159.124:34749)
0927 18:49:19.993453 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.993459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.993470 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:49:19.993494 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.993496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.993524 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.994021 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.994106 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.995928 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.997568 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.997917 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.998394 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:20.000264 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:20.000268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:20.000270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:20.000291 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:20.002124 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:20.002544 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:20.002549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:20.002552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:20.002639 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:20.002986 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:20.002992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:20.002995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:20.003210 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:20.003310 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:20.003660 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 18:49:20.003780 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":111,"threads_started":1}
I20260927 18:49:20.003962 11381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61: Bootstrap starting.
I20260927 18:49:20.003839 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.993021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38344)
0927 18:49:19.993110 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.993125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:19.993191 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:19.993668 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:19.993670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:19.993840 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:19.993925 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.993930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.996016 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.996020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.996872 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.996881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.998071 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.998075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:19.998172 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:19.998602 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:19.998616 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:20.000127 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:20.002253 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:20.002256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:20.002258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:20.002444 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:20.002762 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:20.002765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:20.002767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:20.002871 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:20.003320 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:20.003322 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:20.003464 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 18:49:20.003575 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 18:49:20.003704 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 18:49:20.006973 11381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:20.007988 11381 log.cc:824] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:20.008137 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.997194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:34749 (local address 127.0.0.1:38352)
0927 18:49:19.997364 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:19.997377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:19.997451 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:19.998104 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:19.998109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:19.998310 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:19.998424 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:19.998431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.999573 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:19.999580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:20.000392 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:20.000403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:20.001816 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:20.001822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:20.001925 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:20.002603 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:20.002616 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:20.004273 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:20.006583 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:20.006586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:20.006588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:20.006826 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:20.007134 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:20.007137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:20.007138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:20.007234 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:20.007647 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:20.007650 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:20.007740 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 18:49:20.007845 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 18:49:20.007970 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 18:49:20.008620 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:49:20.008868 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:19.997138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.29.159.124:34749)
0927 18:49:19.997846 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:19.997853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:19.997864 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:49:19.997893 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:19.997898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:19.997936 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:19.998077 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:19.998602 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:19.999420 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:20.001022 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:20.001681 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:20.002123 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:20.004531 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:20.004534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:20.004537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:20.004558 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:20.006434 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:20.006931 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:20.006934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:20.006936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:20.007020 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:20.007328 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:20.007331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:20.007333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:20.007533 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:20.008524 (+ 991us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:20.008593 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 18:49:20.008696 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":96,"threads_started":1}
I20260927 18:49:20.011063 11305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162: Bootstrap starting.
I20260927 18:49:20.013806 11305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:20.016781 11230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8: Bootstrap starting.
I20260927 18:49:20.019706 11230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:20.020057 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:20.010728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37982 (local address 127.29.159.125:39833)
0927 18:49:20.010856 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:20.010860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:20.011004 (+ 144us) server_negotiation.cc:407] Connection header received
0927 18:49:20.011183 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:20.011186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:20.011239 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:20.011285 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:20.012359 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:20.013075 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:20.014173 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:20.014488 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:20.014575 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:20.016359 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:20.016363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:20.016366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:20.016393 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:20.018060 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:20.018561 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:20.018564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:20.018566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:20.018644 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:20.018933 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:20.018936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:20.018938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:20.019140 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:20.019303 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:20.019713 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 18:49:20.019913 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:49:20.020823 11305 log.cc:824] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:20.021713 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:20.011073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.29.159.126:45525)
0927 18:49:20.011189 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:20.011195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:20.011803 (+ 608us) server_negotiation.cc:407] Connection header received
0927 18:49:20.011982 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:20.011985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:20.012027 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:20.012077 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:20.013378 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:20.014141 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:20.015572 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:20.015918 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:20.015988 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:20.017777 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:20.017783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:20.017789 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:20.017821 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:20.019740 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:20.020279 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:20.020285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:20.020287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:20.020376 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:20.020715 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:20.020718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:20.020722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:20.020942 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:20.021041 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:20.021438 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 18:49:20.021557 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:49:20.021719 11230 log.cc:824] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:20.022975 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:20.011531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.29.159.124:34749)
0927 18:49:20.011628 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:20.011632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:20.011848 (+ 216us) server_negotiation.cc:407] Connection header received
0927 18:49:20.011998 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:20.012001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:20.012038 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:20.012085 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:20.013368 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:20.014157 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:20.015755 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:20.016083 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:20.016151 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:20.019040 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:20.019046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:20.019050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:20.019084 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:20.020926 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:20.021438 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:20.021442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:20.021445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:20.021527 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:20.021855 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:20.021859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:20.021861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:20.022075 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:20.022211 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:20.022690 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 18:49:20.022842 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 18:49:20.023685 11381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61: No bootstrap required, opened a new log
I20260927 18:49:20.023707 11305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162: No bootstrap required, opened a new log
I20260927 18:49:20.028043 11305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } }
I20260927 18:49:20.028298 11305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:20.028350 11305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43acae53a4014ef38ad753604c3ab162, State: Initialized, Role: FOLLOWER
I20260927 18:49:20.028600 11381 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } }
I20260927 18:49:20.028832 11381 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:20.028882 11381 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a71b71ab5a4fedbd042ed49d53cb61, State: Initialized, Role: FOLLOWER
I20260927 18:49:20.028836 11305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } }
I20260927 18:49:20.029359 11399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } } }
I20260927 18:49:20.029347 11381 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } }
I20260927 18:49:20.029516 11399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:49:20.029819 11400 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } } }
I20260927 18:49:20.029948 11400 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:49:20.030200 11305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:20.030604 11381 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:20.033402 11230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8: No bootstrap required, opened a new log
W20260927 18:49:20.035992 11422 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:49:20.035992 11417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:49:20.036067 11417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 18:49:20.036154 11422 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:49:20.038914 11230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } }
I20260927 18:49:20.039318 11230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:20.039448 11230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b33a58cdeec408090356e91b24131d8, State: Initialized, Role: FOLLOWER
I20260927 18:49:20.035681 11237 master_runner.cc:186] Error getting master registration for 127.29.159.126:45525: OK, instance_id { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" instance_seqno: 1790534959527958 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:49:20.039598 11237 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:49:20.039979 11230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } }
I20260927 18:49:20.036957 11312 master_runner.cc:186] Error getting master registration for 127.29.159.126:45525: OK, instance_id { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" instance_seqno: 1790534959527958 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:49:20.040283 11312 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:49:20.040527 11425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } } }
I20260927 18:49:20.040654 11425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:49:20.041393 11230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:49:20.046638 11436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:49:20.046787 11436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:49:20.049994 11162 master_runner.cc:263] No leader master found from master 6b33a58cdeec408090356e91b24131d8
I20260927 18:49:20.050056 11162 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:49:20.165822 11399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:20.166018 11399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } }
I20260927 18:49:20.167055 11399 leader_election.cc:288] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b33a58cdeec408090356e91b24131d8 (127.29.159.126:45525), c3a71b71ab5a4fedbd042ed49d53cb61 (127.29.159.124:34749)
I20260927 18:49:20.168009 11204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43acae53a4014ef38ad753604c3ab162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b33a58cdeec408090356e91b24131d8" is_pre_election: true
I20260927 18:49:20.168356 11204 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43acae53a4014ef38ad753604c3ab162 in term 0.
I20260927 18:49:20.168968 11256 leader_election.cc:302] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43acae53a4014ef38ad753604c3ab162, 6b33a58cdeec408090356e91b24131d8; no voters:
I20260927 18:49:20.169358 11399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:49:20.169450 11399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:49:20.169499 11399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:20.170790 11399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } }
I20260927 18:49:20.171126 11399 leader_election.cc:288] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [CANDIDATE]: Term 1 election: Requested vote from peers 6b33a58cdeec408090356e91b24131d8 (127.29.159.126:45525), c3a71b71ab5a4fedbd042ed49d53cb61 (127.29.159.124:34749)
I20260927 18:49:20.171667 11204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43acae53a4014ef38ad753604c3ab162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b33a58cdeec408090356e91b24131d8"
I20260927 18:49:20.171788 11204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:20.173065 11204 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43acae53a4014ef38ad753604c3ab162 in term 1.
I20260927 18:49:20.173532 11256 leader_election.cc:302] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43acae53a4014ef38ad753604c3ab162, 6b33a58cdeec408090356e91b24131d8; no voters:
I20260927 18:49:20.174171 11355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43acae53a4014ef38ad753604c3ab162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" is_pre_election: true
I20260927 18:49:20.174494 11355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43acae53a4014ef38ad753604c3ab162 in term 0.
I20260927 18:49:20.174773 11354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43acae53a4014ef38ad753604c3ab162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61"
I20260927 18:49:20.174880 11354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:20.176074 11354 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43acae53a4014ef38ad753604c3ab162 in term 1.
I20260927 18:49:20.176815 11399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:20.177100 11399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [term 1 LEADER]: Becoming Leader. State: Replica: 43acae53a4014ef38ad753604c3ab162, State: Running, Role: LEADER
I20260927 18:49:20.177466 11399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } }
I20260927 18:49:20.179136 11399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43acae53a4014ef38ad753604c3ab162. Latest consensus state: current_term: 1 leader_uuid: "43acae53a4014ef38ad753604c3ab162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } } }
I20260927 18:49:20.180962 11399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:20.183449 11440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:20.184444 11440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:20.189811 11354 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [term 1 FOLLOWER]: Refusing update from remote peer 43acae53a4014ef38ad753604c3ab162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:20.190573 11438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:20.194852 11400 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43acae53a4014ef38ad753604c3ab162. Latest consensus state: current_term: 1 leader_uuid: "43acae53a4014ef38ad753604c3ab162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } } }
I20260927 18:49:20.194991 11400 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:49:20.196210 11204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [term 1 FOLLOWER]: Refusing update from remote peer 43acae53a4014ef38ad753604c3ab162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:20.198971 11438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:20.202946 11425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43acae53a4014ef38ad753604c3ab162. Latest consensus state: current_term: 1 leader_uuid: "43acae53a4014ef38ad753604c3ab162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } } }
I20260927 18:49:20.203064 11425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 18:49:20.203919 11334 tablet.cc:2369] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:20.208189 11438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43acae53a4014ef38ad753604c3ab162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } } }
I20260927 18:49:20.208323 11438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:20.209270 11441 mvcc.cc:205] Tried to move back new op lower bound from 7334031196930203648 to 7334031196889669632. Current Snapshot: MvccSnapshot[applied={T|T < 7334031196930203648}]
I20260927 18:49:20.209812 11425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43acae53a4014ef38ad753604c3ab162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } } }
I20260927 18:49:20.209914 11425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:49:20.210431 11444 mvcc.cc:205] Tried to move back new op lower bound from 7334031196930203648 to 7334031196889669632. Current Snapshot: MvccSnapshot[applied={T|T < 7334031196930203648}]
I20260927 18:49:20.211577 11399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43acae53a4014ef38ad753604c3ab162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } } }
I20260927 18:49:20.211676 11399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:20.212587 11400 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43acae53a4014ef38ad753604c3ab162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b33a58cdeec408090356e91b24131d8" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 45525 } } peers { permanent_uuid: "43acae53a4014ef38ad753604c3ab162" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 39833 } } peers { permanent_uuid: "c3a71b71ab5a4fedbd042ed49d53cb61" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 34749 } } }
I20260927 18:49:20.212703 11400 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:49:20.212819 11442 mvcc.cc:205] Tried to move back new op lower bound from 7334031196930203648 to 7334031196889669632. Current Snapshot: MvccSnapshot[applied={T|T < 7334031196930203648}]
I20260927 18:49:20.214346 11440 catalog_manager.cc:1347] Generated new cluster ID: 5251f0f4c6f740b981b385d9e1a0bf37
I20260927 18:49:20.214402 11440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:20.250509 11440 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:20.251866 11440 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:20.264408 11440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43acae53a4014ef38ad753604c3ab162: Generated new TSK 0
I20260927 18:49:20.265373 11440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:49:21.037465 11422 catalog_manager.cc:1259] Loaded cluster ID: 5251f0f4c6f740b981b385d9e1a0bf37
I20260927 18:49:21.037547 11422 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61: loading cluster ID for follower catalog manager: success
I20260927 18:49:21.039261 11422 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61: acquiring CA information for follower catalog manager: success
I20260927 18:49:21.040460 11422 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c3a71b71ab5a4fedbd042ed49d53cb61: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:49:21.047948 11436 catalog_manager.cc:1259] Loaded cluster ID: 5251f0f4c6f740b981b385d9e1a0bf37
I20260927 18:49:21.048008 11436 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8: loading cluster ID for follower catalog manager: success
I20260927 18:49:21.049626 11436 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8: acquiring CA information for follower catalog manager: success
I20260927 18:49:21.050720 11436 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b33a58cdeec408090356e91b24131d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:49:21.334026 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 11162
I20260927 18:49:21.340238 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 11237
I20260927 18:49:21.345916 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 11312
2026-09-27T18:49:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3181 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:49:21 dist-test-slave-8tg2 krb5kdc[11455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:49:21 dist-test-slave-8tg2 krb5kdc[11455](info): set up 2 sockets
Sep 27 18:49:21 dist-test-slave-8tg2 krb5kdc[11455](info): commencing operation
krb5kdc: starting...
W20260927 18:49:22.407780 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:49:22 dist-test-slave-8tg2 krb5kdc[11455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:49:22.530658 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:38409
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:38409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:22.640976 11468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:22.641276 11468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:22.641320 11468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:22.646164 11468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:22.646252 11468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:22.646276 11468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:22.646296 11468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:22.646313 11468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:22.652060 11468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:38409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:38409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:22.653052 11468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:22.654054 11468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:22.662598 11476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:22.662693 11474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:22.662600 11473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:22.662922 11468 server_base.cc:1034] running on GCE node
I20260927 18:49:22.663570 11468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:22.664669 11468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:22.665835 11468 hybrid_clock.cc:648] HybridClock initialized: now 1790534962665817 us; error 32 us; skew 500 ppm
Sep 27 18:49:22 dist-test-slave-8tg2 krb5kdc[11455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534962, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:22.675405 11468 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:49:22.676460 11468 webserver.cc:460] Webserver started at http://127.29.159.126:40349/ using document root <none> and password file <none>
I20260927 18:49:22.677006 11468 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:22.677063 11468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:22.677284 11468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:22.679376 11468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "b36221b005e54aa0b2914df3623687a6"
format_stamp: "Formatted at 2026-09-27 18:49:22 on dist-test-slave-8tg2"
server_key: "76bdd97aa34b09aebbf583b6cadd8f2b"
server_key_iv: "8d7e91d9ac3615bd6228084087d762bb"
server_key_version: "encryptionkey@0"
I20260927 18:49:22.679971 11468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "b36221b005e54aa0b2914df3623687a6"
format_stamp: "Formatted at 2026-09-27 18:49:22 on dist-test-slave-8tg2"
server_key: "76bdd97aa34b09aebbf583b6cadd8f2b"
server_key_iv: "8d7e91d9ac3615bd6228084087d762bb"
server_key_version: "encryptionkey@0"
I20260927 18:49:22.683782 11468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:49:22.686342 11483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:22.687671 11468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:22.687805 11468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "b36221b005e54aa0b2914df3623687a6"
format_stamp: "Formatted at 2026-09-27 18:49:22 on dist-test-slave-8tg2"
server_key: "76bdd97aa34b09aebbf583b6cadd8f2b"
server_key_iv: "8d7e91d9ac3615bd6228084087d762bb"
server_key_version: "encryptionkey@0"
I20260927 18:49:22.687924 11468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:22.706120 11468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:22.709653 11468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:22.709861 11468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:22.717532 11468 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:38409
I20260927 18:49:22.717572 11535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:38409 every 8 connection(s)
I20260927 18:49:22.718933 11468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:22.722970 11536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:22.727015 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 11468
I20260927 18:49:22.727106 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:49:22.727408 30333 external_mini_cluster.cc:1428] Setting key 5c97f3508961238491dfa99ce0f7a501
I20260927 18:49:22.729063 11536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6: Bootstrap starting.
I20260927 18:49:22.731769 11536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:22.732648 11536 log.cc:824] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:22.734455 11536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6: No bootstrap required, opened a new log
Sep 27 18:49:22 dist-test-slave-8tg2 krb5kdc[11455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:22.739267 11536 raft_consensus.cc:348] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36221b005e54aa0b2914df3623687a6" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38409 } }
I20260927 18:49:22.739564 11536 raft_consensus.cc:374] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:22.739629 11536 raft_consensus.cc:729] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36221b005e54aa0b2914df3623687a6, State: Initialized, Role: FOLLOWER
I20260927 18:49:22.740082 11536 consensus_queue.cc:260] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36221b005e54aa0b2914df3623687a6" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38409 } }
I20260927 18:49:22.740224 11536 raft_consensus.cc:388] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:22.740314 11536 raft_consensus.cc:482] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:22.740413 11536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:22.741248 11536 raft_consensus.cc:504] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36221b005e54aa0b2914df3623687a6" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38409 } }
I20260927 18:49:22.741637 11536 leader_election.cc:302] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b36221b005e54aa0b2914df3623687a6; no voters:
I20260927 18:49:22.741936 11536 leader_election.cc:288] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:49:22.742054 11541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:22.742242 11541 raft_consensus.cc:686] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [term 1 LEADER]: Becoming Leader. State: Replica: b36221b005e54aa0b2914df3623687a6, State: Running, Role: LEADER
I20260927 18:49:22.742424 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:22.728825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.29.159.126:38409)
0927 18:49:22.729234 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:22.729253 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:22.729296 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:49:22.730092 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:22.730114 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:22.730555 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:22.730917 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:22.731658 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:22.733172 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:22.734039 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:22.734514 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:22.734682 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:22.737398 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:22.737408 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:22.737427 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:22.737464 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:22.739459 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:22.739956 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:22.739962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:22.739973 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:22.740080 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:22.740348 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:22.740352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:22.740354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:22.740716 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:22.740954 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:22.741186 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 18:49:22.741493 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":107,"threads_started":1}
I20260927 18:49:22.742594 11541 consensus_queue.cc:237] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36221b005e54aa0b2914df3623687a6" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38409 } }
I20260927 18:49:22.743134 11536 sys_catalog.cc:564] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:22.743721 11543 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b36221b005e54aa0b2914df3623687a6. Latest consensus state: current_term: 1 leader_uuid: "b36221b005e54aa0b2914df3623687a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36221b005e54aa0b2914df3623687a6" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38409 } } }
I20260927 18:49:22.743916 11543 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:22.744153 11542 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b36221b005e54aa0b2914df3623687a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36221b005e54aa0b2914df3623687a6" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38409 } } }
I20260927 18:49:22.744239 11542 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:22.744727 11550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:22.745991 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:49:22.748054 11550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:22.753213 11550 catalog_manager.cc:1347] Generated new cluster ID: 26f49ac6477841e5a34b4f7754e97609
I20260927 18:49:22.753279 11550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:22.775110 11550 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:22.776049 11550 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:22.786041 11550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b36221b005e54aa0b2914df3623687a6: Generated new TSK 0
I20260927 18:49:22.786702 11550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:49:22.801608 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 11468
2026-09-27T18:49:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1453 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:49:22 dist-test-slave-8tg2 krb5kdc[11563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:49:22 dist-test-slave-8tg2 krb5kdc[11563](info): set up 2 sockets
Sep 27 18:49:22 dist-test-slave-8tg2 krb5kdc[11563](info): commencing operation
krb5kdc: starting...
W20260927 18:49:23.843498 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:49:23 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:49:23.967192 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41559
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:41559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:24.077847 11576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:24.078229 11576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:24.078336 11576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:24.082463 11576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:24.082552 11576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:24.082590 11576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:24.082646 11576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:24.082695 11576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:24.088167 11576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:41559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:24.089250 11576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:24.090289 11576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:24.098737 11581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:24.098848 11576 server_base.cc:1034] running on GCE node
W20260927 18:49:24.098753 11584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:24.098737 11582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:24.099512 11576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:24.100593 11576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:24.101759 11576 hybrid_clock.cc:648] HybridClock initialized: now 1790534964101738 us; error 32 us; skew 500 ppm
Sep 27 18:49:24 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534964, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:24.104669 11576 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:49:24.105479 11576 webserver.cc:460] Webserver started at http://127.29.159.126:40475/ using document root <none> and password file <none>
I20260927 18:49:24.106019 11576 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:24.106074 11576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:24.106336 11576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:24.108433 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "d84c9a30434541899b41a22b20dfc13f"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "534f3ac80b64063cebe0c757902bfa37"
server_key_iv: "d3bd5be9b17fc7b65bacfa9e8d05a4fe"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.108985 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "d84c9a30434541899b41a22b20dfc13f"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "534f3ac80b64063cebe0c757902bfa37"
server_key_iv: "d3bd5be9b17fc7b65bacfa9e8d05a4fe"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.112728 11576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:49:24.115104 11591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:24.116446 11576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:49:24.116586 11576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "d84c9a30434541899b41a22b20dfc13f"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "534f3ac80b64063cebe0c757902bfa37"
server_key_iv: "d3bd5be9b17fc7b65bacfa9e8d05a4fe"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.116698 11576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:24.128276 11576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:24.131551 11576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:24.131749 11576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:24.138479 11576 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:41559
I20260927 18:49:24.138790 11643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:41559 every 8 connection(s)
I20260927 18:49:24.139583 11576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:24.142318 11644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:24.143399 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 11576
I20260927 18:49:24.143548 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:49:24.143901 30333 external_mini_cluster.cc:1428] Setting key 796510e2214e2c16c1caed7dba01d01d
I20260927 18:49:24.148690 11644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Bootstrap starting.
I20260927 18:49:24.150875 11644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:24.151625 11644 log.cc:824] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:49:24 dist-test-slave-8tg2 krb5kdc[11563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:24.153460 11644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: No bootstrap required, opened a new log
I20260927 18:49:24.156858 11644 raft_consensus.cc:348] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:24.157088 11644 raft_consensus.cc:374] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:24.157162 11644 raft_consensus.cc:729] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84c9a30434541899b41a22b20dfc13f, State: Initialized, Role: FOLLOWER
I20260927 18:49:24.157579 11647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.145216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.29.159.126:41559)
0927 18:49:24.145611 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:24.145627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:24.145660 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:49:24.146257 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:24.146272 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:24.146583 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:24.146848 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:24.147792 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.148743 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.149676 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.150151 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.150339 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.152906 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:24.152911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:24.152923 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:24.152949 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:24.154708 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:24.155143 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:24.155149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:24.155161 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:24.155268 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:24.155592 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:24.155595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:24.155597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:24.155980 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:24.156193 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:24.156469 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 18:49:24.156740 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":138,"threads_started":1}
I20260927 18:49:24.157584 11644 consensus_queue.cc:260] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:24.157815 11644 raft_consensus.cc:388] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:24.157898 11644 raft_consensus.cc:482] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:24.157996 11644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:24.158999 11644 raft_consensus.cc:504] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:24.159348 11644 leader_election.cc:302] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d84c9a30434541899b41a22b20dfc13f; no voters:
I20260927 18:49:24.159683 11644 leader_election.cc:288] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:49:24.159788 11649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:24.160017 11649 raft_consensus.cc:686] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 1 LEADER]: Becoming Leader. State: Replica: d84c9a30434541899b41a22b20dfc13f, State: Running, Role: LEADER
I20260927 18:49:24.160323 11649 consensus_queue.cc:237] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:24.160920 11644 sys_catalog.cc:564] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:24.162027 11651 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d84c9a30434541899b41a22b20dfc13f. Latest consensus state: current_term: 1 leader_uuid: "d84c9a30434541899b41a22b20dfc13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } } }
I20260927 18:49:24.162006 11650 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d84c9a30434541899b41a22b20dfc13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } } }
I20260927 18:49:24.162147 11650 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:24.162147 11651 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:24.162581 11658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:24.165664 11658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:24.170795 11658 catalog_manager.cc:1347] Generated new cluster ID: c623aa9f592249b1b987dfdee4794ba8
I20260927 18:49:24.170861 11658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:24.182165 11658 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:24.183187 11658 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:24.187512 11658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Generated new TSK 0
I20260927 18:49:24.188275 11658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:49:24.218158 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41559
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:24.329149 11670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:24.329409 11670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:24.329452 11670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:24.333540 11670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:24.333604 11670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:24.333662 11670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:49:24.338930 11670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:24.340114 11670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:24.341147 11670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:24.350924 11678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:24.351035 11670 server_base.cc:1034] running on GCE node
W20260927 18:49:24.350879 11676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:24.350945 11675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:24.351603 11670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:24.352243 11670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:24.353452 11670 hybrid_clock.cc:648] HybridClock initialized: now 1790534964353420 us; error 44 us; skew 500 ppm
Sep 27 18:49:24 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534964, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:24.356254 11670 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:49:24.357061 11670 webserver.cc:460] Webserver started at http://127.29.159.65:32987/ using document root <none> and password file <none>
I20260927 18:49:24.357556 11670 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:24.357611 11670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:24.357766 11670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:24.359745 11670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "19bfd55a6c764979a211402015c5dacb"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "1380b6db95c0a0ded47ae7d6aa3b6d78"
server_key_iv: "6aecb2743ce48fb12f0c1f9e0f70599a"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.360217 11670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "19bfd55a6c764979a211402015c5dacb"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "1380b6db95c0a0ded47ae7d6aa3b6d78"
server_key_iv: "6aecb2743ce48fb12f0c1f9e0f70599a"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.364046 11670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:49:24.366228 11685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:24.367399 11670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:49:24.367544 11670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "19bfd55a6c764979a211402015c5dacb"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "1380b6db95c0a0ded47ae7d6aa3b6d78"
server_key_iv: "6aecb2743ce48fb12f0c1f9e0f70599a"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.367640 11670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:24.381148 11670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:24.384392 11670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:24.384547 11670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:24.385030 11670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:24.385789 11670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:24.385834 11670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:24.385874 11670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:24.385890 11670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:24.394798 11670 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:46481
I20260927 18:49:24.394824 11798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:46481 every 8 connection(s)
I20260927 18:49:24.395768 11670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:24.404548 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 11670
I20260927 18:49:24.404685 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:49:24.405073 30333 external_mini_cluster.cc:1428] Setting key 39aa9cf1bfea8af4fe50cdfc80114752
Sep 27 18:49:24 dist-test-slave-8tg2 krb5kdc[11563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534964, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:24.410040 11647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.397668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:33791 (local address 127.29.159.126:41559)
0927 18:49:24.397813 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:24.397817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:24.398642 (+ 825us) server_negotiation.cc:407] Connection header received
0927 18:49:24.400030 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:24.400033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:24.400088 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:24.400195 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:24.401435 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.401984 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.403151 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.403385 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.403551 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.406365 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:24.406368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:24.406370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:24.406392 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:24.407884 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:24.408405 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:24.408409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:24.408410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:24.408490 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:24.408841 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:24.408844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:24.408845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:24.409022 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:24.409132 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:24.409718 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 18:49:24.409923 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:49:24.411182 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.398025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:33791)
0927 18:49:24.398488 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:24.398528 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:24.399820 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:24.400366 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:24.400373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:24.400806 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:24.401288 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.401302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.402109 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.402113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.403024 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.403035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.403584 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.403588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.403753 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.404350 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:24.404397 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:24.406157 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:24.408000 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:24.408006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:24.408020 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:24.408276 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:24.408592 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:24.408595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:24.408597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:24.408740 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:24.409130 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:24.409137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:24.409453 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 18:49:24.409667 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 18:49:24.410052 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260927 18:49:24.412524 11799 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41559
I20260927 18:49:24.412762 11799 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:24.413317 11799 heartbeater.cc:511] Master 127.29.159.126:41559 requested a full tablet report, sending...
I20260927 18:49:24.414837 11608 ts_manager.cc:194] Registered new tserver with Master: 19bfd55a6c764979a211402015c5dacb (127.29.159.65:46481)
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:49:24.416044 11608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:33791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:49:24.434276 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41559
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:24.542083 11804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:24.542383 11804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:24.542459 11804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:24.546413 11804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:24.546497 11804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:24.546584 11804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:49:24.551838 11804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:24.552834 11804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:24.553826 11804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:24.563527 11809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:24.563516 11810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:24.563516 11812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:24.563863 11804 server_base.cc:1034] running on GCE node
I20260927 18:49:24.564239 11804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:24.564903 11804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:24.566052 11804 hybrid_clock.cc:648] HybridClock initialized: now 1790534964566024 us; error 44 us; skew 500 ppm
Sep 27 18:49:24 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534964, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:24.568770 11804 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:49:24.569597 11804 webserver.cc:460] Webserver started at http://127.29.159.66:39223/ using document root <none> and password file <none>
I20260927 18:49:24.570161 11804 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:24.570284 11804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:24.570518 11804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:24.572554 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "88d6b57e91a549d196c385819186d09b"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "42b9be8a1814e03539e585cc728a2e76"
server_key_iv: "b8c434d16e111f2b82844b5c216b7717"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.573114 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "88d6b57e91a549d196c385819186d09b"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "42b9be8a1814e03539e585cc728a2e76"
server_key_iv: "b8c434d16e111f2b82844b5c216b7717"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.576721 11804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:49:24.578998 11819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:24.580209 11804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:49:24.580355 11804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "88d6b57e91a549d196c385819186d09b"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "42b9be8a1814e03539e585cc728a2e76"
server_key_iv: "b8c434d16e111f2b82844b5c216b7717"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.580476 11804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:24.608464 11804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:24.611927 11804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:24.612131 11804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:24.612766 11804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:24.613638 11804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:24.613715 11804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:24.613777 11804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:24.613828 11804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:24.622925 11804 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:33857
I20260927 18:49:24.622977 11932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:33857 every 8 connection(s)
I20260927 18:49:24.623862 11804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:24.630548 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 11804
I20260927 18:49:24.630666 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:49:24.630956 30333 external_mini_cluster.cc:1428] Setting key 689394a0323eca1f13cfafe658a0045c
Sep 27 18:49:24 dist-test-slave-8tg2 krb5kdc[11563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534964, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:24.638383 11647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.625660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:37339 (local address 127.29.159.126:41559)
0927 18:49:24.625799 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:24.625803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:24.626625 (+ 822us) server_negotiation.cc:407] Connection header received
0927 18:49:24.628053 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:24.628057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:24.628114 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:24.628220 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:24.629352 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.629911 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.631490 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.631806 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.631922 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.634461 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:24.634467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:24.634472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:24.634505 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:24.636155 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:24.636666 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:24.636670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:24.636672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:24.636771 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:24.637129 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:24.637136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:24.637139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:24.637402 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:24.637513 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:24.637978 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 18:49:24.638122 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:49:24.639045 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.626049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:37339)
0927 18:49:24.626484 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:24.626523 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:24.627832 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:24.628390 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:24.628398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:24.628746 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:24.629210 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.629223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.630045 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.630053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.631347 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.631356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.631957 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.631961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.632124 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.632698 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:24.632714 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:24.634287 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:24.636267 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:24.636272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:24.636283 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:24.636558 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:24.636889 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:24.636892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:24.636893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:24.637034 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:24.637510 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:24.637515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:24.637731 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 18:49:24.637898 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 18:49:24.638146 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260927 18:49:24.640242 11933 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41559
I20260927 18:49:24.640470 11933 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:49:24.640995 11933 heartbeater.cc:511] Master 127.29.159.126:41559 requested a full tablet report, sending...
I20260927 18:49:24.642017 11608 ts_manager.cc:194] Registered new tserver with Master: 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857)
I20260927 18:49:24.642575 11608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:37339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:49:24.659145 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41559
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:24.776259 11938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:24.776499 11938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:24.776540 11938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:24.780684 11938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:24.780741 11938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:24.780797 11938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:49:24.786108 11938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:24.787133 11938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:24.788254 11938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:24.797822 11943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:24.797830 11946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:24.797832 11944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:24.797937 11938 server_base.cc:1034] running on GCE node
I20260927 18:49:24.798448 11938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:24.799062 11938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:24.800241 11938 hybrid_clock.cc:648] HybridClock initialized: now 1790534964800227 us; error 26 us; skew 500 ppm
Sep 27 18:49:24 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534964, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:24.803053 11938 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:49:24.804088 11938 webserver.cc:460] Webserver started at http://127.29.159.67:34517/ using document root <none> and password file <none>
I20260927 18:49:24.804780 11938 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:24.804845 11938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:24.805068 11938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:24.807029 11938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "88b6e3b4e1c94ffaa9b90561e127b594"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "f8920269a0470bf5d3faa7ad8c4ff6c8"
server_key_iv: "8ef0caf9e2f071e1695f4bb6a60def6d"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.807624 11938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "88b6e3b4e1c94ffaa9b90561e127b594"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "f8920269a0470bf5d3faa7ad8c4ff6c8"
server_key_iv: "8ef0caf9e2f071e1695f4bb6a60def6d"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.811153 11938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:49:24.813530 11953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:24.814810 11938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 18:49:24.814939 11938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "88b6e3b4e1c94ffaa9b90561e127b594"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "f8920269a0470bf5d3faa7ad8c4ff6c8"
server_key_iv: "8ef0caf9e2f071e1695f4bb6a60def6d"
server_key_version: "encryptionkey@0"
I20260927 18:49:24.815037 11938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:24.831219 11938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:24.834687 11938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:24.834900 11938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:24.835565 11938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:24.836385 11938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:24.836436 11938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:24.836508 11938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:24.836545 11938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:24.845665 11938 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:35751
I20260927 18:49:24.845723 12066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:35751 every 8 connection(s)
I20260927 18:49:24.846581 11938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:24.855931 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 11938
I20260927 18:49:24.856029 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:49:24.856323 30333 external_mini_cluster.cc:1428] Setting key d2b828438a6d21dff9d08d87a665dce2
Sep 27 18:49:24 dist-test-slave-8tg2 krb5kdc[11563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534964, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:24.860311 11647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.848584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:56123 (local address 127.29.159.126:41559)
0927 18:49:24.848749 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:24.848752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:24.849607 (+ 855us) server_negotiation.cc:407] Connection header received
0927 18:49:24.851008 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:24.851012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:24.851075 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:24.851181 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:24.852385 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.852919 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.854076 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.854307 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.854420 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.857009 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:24.857013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:24.857015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:24.857048 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:24.858505 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:24.858970 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:24.858972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:24.858973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:24.859045 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:24.859380 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:24.859382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:24.859383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:24.859595 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:24.859673 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:24.860107 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 18:49:24.860199 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:49:24.861200 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.848933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:56123)
0927 18:49:24.849439 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:24.849497 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:24.850789 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:24.851364 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:24.851370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:24.851759 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:24.852230 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.852243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.853054 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.853058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.853948 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.853956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.854454 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.854458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.854631 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.855186 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:24.855201 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:24.856840 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:24.858614 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:24.858618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:24.858630 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:24.858879 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:24.859125 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:24.859127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:24.859128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:24.859249 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:24.859683 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:24.859687 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:24.859900 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 18:49:24.860059 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 18:49:24.860327 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260927 18:49:24.862314 12067 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41559
I20260927 18:49:24.862557 12067 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:24.863030 12067 heartbeater.cc:511] Master 127.29.159.126:41559 requested a full tablet report, sending...
I20260927 18:49:24.864035 11608 ts_manager.cc:194] Registered new tserver with Master: 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751)
I20260927 18:49:24.864557 11608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:56123
I20260927 18:49:24.870428 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:49:24.880394 11647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.872626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.29.159.126:41559)
0927 18:49:24.872767 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:24.872771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:24.873032 (+ 261us) server_negotiation.cc:407] Connection header received
0927 18:49:24.873076 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:24.873078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:24.873118 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:24.873190 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:24.873877 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.874432 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.875362 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.875658 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.875743 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.877175 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:24.877178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:24.877180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:24.877201 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:24.878700 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:24.879122 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:24.879124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:24.879126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:24.879196 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:24.879535 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:24.879537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:24.879538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:24.879743 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:24.879854 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:24.880056 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 18:49:24.880147 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:49:24.883872 11608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:49:24.887189 11608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:49:24.901938 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.896907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.29.159.65:46481)
0927 18:49:24.897277 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:24.897290 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:24.897360 (+ 70us) server_negotiation.cc:407] Connection header received
0927 18:49:24.897443 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:24.897448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:24.897634 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:24.897780 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:24.898065 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.899040 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.900531 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.901221 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.901349 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.901506 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:24.901669 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 18:49:24.901769 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":93,"threads_started":1}
I20260927 18:49:24.901923 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.896761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:46481 (local address 127.0.0.1:38564)
0927 18:49:24.896983 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:24.896998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:24.897076 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:24.897828 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:24.897832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:24.897850 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:24.897939 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.897944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.899176 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.899180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.900390 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.900400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.901348 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.901352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.901461 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.901498 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:49:24.901637 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 18:49:24.901766 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260927 18:49:24.902249 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.896434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:33857 (local address 127.0.0.1:40256)
0927 18:49:24.896680 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:24.896697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:24.896785 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:24.898181 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:24.898184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:24.898213 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:24.898304 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.898310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.899601 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.899605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.900714 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.900722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.901899 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.901902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.901992 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.902015 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:49:24.902084 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:49:24.902143 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
I20260927 18:49:24.902244 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.895839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.29.159.67:35751)
0927 18:49:24.896298 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:24.896315 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:24.896346 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:49:24.896431 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:24.896436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:24.896674 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:24.896859 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:24.897388 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.898592 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.900477 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.901249 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.901356 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.901506 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:24.901992 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 18:49:24.902104 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":139,"threads_started":1}
I20260927 18:49:24.902289 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.895666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:35751 (local address 127.0.0.1:34706)
0927 18:49:24.896047 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:24.896102 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:24.896229 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:24.897072 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:24.897079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:24.897106 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:24.897241 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.897248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.898733 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.898737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.900339 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.900351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.901572 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.901577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.901683 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.901708 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:49:24.901773 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:49:24.901864 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":58,"thread_start_us":90,"threads_started":1}
I20260927 18:49:24.902689 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.897296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.29.159.66:33857)
0927 18:49:24.897740 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:24.897751 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:24.897775 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:49:24.897849 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:24.897854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:24.898046 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:24.898229 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:24.898409 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.899469 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.900835 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.901613 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.901708 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.901841 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:24.902439 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 18:49:24.902571 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":122,"threads_started":1}
I20260927 18:49:24.905383 11867 tablet_service.cc:1467] Processing CreateTablet for tablet 96462d42d08546d5a1b1b1a4287259f8 (DEFAULT_TABLE table=test-table [id=d680a339960e45a2a65d4e84803e671b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:24.905551 12001 tablet_service.cc:1467] Processing CreateTablet for tablet 96462d42d08546d5a1b1b1a4287259f8 (DEFAULT_TABLE table=test-table [id=d680a339960e45a2a65d4e84803e671b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:24.905857 11733 tablet_service.cc:1467] Processing CreateTablet for tablet 96462d42d08546d5a1b1b1a4287259f8 (DEFAULT_TABLE table=test-table [id=d680a339960e45a2a65d4e84803e671b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:24.906474 11867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96462d42d08546d5a1b1b1a4287259f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:24.906628 12001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96462d42d08546d5a1b1b1a4287259f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:24.906765 11733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96462d42d08546d5a1b1b1a4287259f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:24.913682 12084 tablet_bootstrap.cc:492] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b: Bootstrap starting.
I20260927 18:49:24.914546 12085 tablet_bootstrap.cc:492] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594: Bootstrap starting.
I20260927 18:49:24.915107 12086 tablet_bootstrap.cc:492] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Bootstrap starting.
I20260927 18:49:24.916852 12085 tablet_bootstrap.cc:654] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:24.917042 12086 tablet_bootstrap.cc:654] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:24.917045 12084 tablet_bootstrap.cc:654] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:24.917642 12085 log.cc:824] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:24.917870 12084 log.cc:824] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:24.917870 12086 log.cc:824] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:24.920020 12084 tablet_bootstrap.cc:492] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b: No bootstrap required, opened a new log
I20260927 18:49:24.920213 12084 ts_tablet_manager.cc:1397] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:49:24.920284 12086 tablet_bootstrap.cc:492] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: No bootstrap required, opened a new log
I20260927 18:49:24.920477 12086 ts_tablet_manager.cc:1397] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:49:24.924265 12084 raft_consensus.cc:348] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:24.924554 12084 raft_consensus.cc:374] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:24.924618 12084 raft_consensus.cc:729] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d6b57e91a549d196c385819186d09b, State: Initialized, Role: FOLLOWER
I20260927 18:49:24.924619 12085 tablet_bootstrap.cc:492] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594: No bootstrap required, opened a new log
I20260927 18:49:24.924840 12085 ts_tablet_manager.cc:1397] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260927 18:49:24.924692 12086 raft_consensus.cc:348] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:24.924926 12086 raft_consensus.cc:374] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:24.924973 12086 raft_consensus.cc:729] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bfd55a6c764979a211402015c5dacb, State: Initialized, Role: FOLLOWER
I20260927 18:49:24.925117 12084 consensus_queue.cc:260] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:24.925350 12086 consensus_queue.cc:260] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:24.926160 12084 ts_tablet_manager.cc:1428] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:49:24.926160 12086 ts_tablet_manager.cc:1428] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:49:24.926502 11799 heartbeater.cc:503] Master 127.29.159.126:41559 was elected leader, sending a full tablet report...
I20260927 18:49:24.926499 11933 heartbeater.cc:503] Master 127.29.159.126:41559 was elected leader, sending a full tablet report...
I20260927 18:49:24.928428 12085 raft_consensus.cc:348] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:24.928610 12085 raft_consensus.cc:374] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:24.928690 12085 raft_consensus.cc:729] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b6e3b4e1c94ffaa9b90561e127b594, State: Initialized, Role: FOLLOWER
I20260927 18:49:24.929140 12085 consensus_queue.cc:260] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:24.930074 12067 heartbeater.cc:503] Master 127.29.159.126:41559 was elected leader, sending a full tablet report...
I20260927 18:49:24.930164 12085 ts_tablet_manager.cc:1428] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:49:24.954375 12090 raft_consensus.cc:482] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:24.954610 12090 raft_consensus.cc:504] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:24.955824 12090 leader_election.cc:288] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751), 19bfd55a6c764979a211402015c5dacb (127.29.159.65:46481)
I20260927 18:49:24.960197 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.956127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:35751 (local address 127.29.159.66:54799)
0927 18:49:24.956293 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:24.956312 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:24.956405 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:24.956795 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:24.956798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:24.956817 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:24.956972 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.956980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.957911 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.957915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.958897 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.958906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.959735 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.959739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.959850 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.959875 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:49:24.959966 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 18:49:24.960052 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 18:49:24.960424 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.956223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:54799 (local address 127.29.159.67:35751)
0927 18:49:24.956386 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:24.956392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:24.956409 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:49:24.956547 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:24.956551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:24.956601 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:24.956667 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:24.957081 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.957749 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.959023 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.959602 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.959701 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.959743 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:24.960179 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 18:49:24.960270 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 18:49:24.960469 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.956408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:46481 (local address 127.29.159.66:54751)
0927 18:49:24.956858 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:24.956871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:24.956983 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:24.957229 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:24.957232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:24.957250 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:24.957367 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.957374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.958347 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.958349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.959202 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:24.959213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.960071 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.960076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:24.960171 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.960196 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:49:24.960273 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:49:24.960335 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":1,"thread_start_us":129,"threads_started":1}
I20260927 18:49:24.960554 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.956612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:54751 (local address 127.29.159.65:46481)
0927 18:49:24.956734 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:24.956739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:24.956967 (+ 228us) server_negotiation.cc:407] Connection header received
0927 18:49:24.957077 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:24.957082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:24.957130 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:24.957197 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:24.957514 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.958216 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.959357 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.959895 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.960029 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.960074 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:24.960316 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 18:49:24.960404 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:49:24.961012 12021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96462d42d08546d5a1b1b1a4287259f8" candidate_uuid: "88d6b57e91a549d196c385819186d09b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" is_pre_election: true
I20260927 18:49:24.961320 12021 raft_consensus.cc:2454] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88d6b57e91a549d196c385819186d09b in term 0.
I20260927 18:49:24.961840 11822 leader_election.cc:302] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b6e3b4e1c94ffaa9b90561e127b594, 88d6b57e91a549d196c385819186d09b; no voters:
I20260927 18:49:24.961896 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96462d42d08546d5a1b1b1a4287259f8" candidate_uuid: "88d6b57e91a549d196c385819186d09b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bfd55a6c764979a211402015c5dacb" is_pre_election: true
I20260927 18:49:24.962217 11753 raft_consensus.cc:2454] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88d6b57e91a549d196c385819186d09b in term 0.
I20260927 18:49:24.962199 12090 raft_consensus.cc:2781] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:49:24.962301 12090 raft_consensus.cc:482] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:49:24.962426 12090 raft_consensus.cc:3037] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:24.963572 12090 raft_consensus.cc:504] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:24.963871 12090 leader_election.cc:288] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [CANDIDATE]: Term 1 election: Requested vote from peers 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751), 19bfd55a6c764979a211402015c5dacb (127.29.159.65:46481)
I20260927 18:49:24.964290 11753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96462d42d08546d5a1b1b1a4287259f8" candidate_uuid: "88d6b57e91a549d196c385819186d09b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bfd55a6c764979a211402015c5dacb"
I20260927 18:49:24.964332 12021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96462d42d08546d5a1b1b1a4287259f8" candidate_uuid: "88d6b57e91a549d196c385819186d09b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b6e3b4e1c94ffaa9b90561e127b594"
I20260927 18:49:24.964434 11753 raft_consensus.cc:3037] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:24.964481 12021 raft_consensus.cc:3037] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:24.965495 11753 raft_consensus.cc:2454] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88d6b57e91a549d196c385819186d09b in term 1.
I20260927 18:49:24.965569 12021 raft_consensus.cc:2454] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88d6b57e91a549d196c385819186d09b in term 1.
I20260927 18:49:24.965816 11822 leader_election.cc:302] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bfd55a6c764979a211402015c5dacb, 88d6b57e91a549d196c385819186d09b; no voters:
I20260927 18:49:24.966017 12090 raft_consensus.cc:2781] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:24.966252 12090 raft_consensus.cc:686] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 1 LEADER]: Becoming Leader. State: Replica: 88d6b57e91a549d196c385819186d09b, State: Running, Role: LEADER
I20260927 18:49:24.966598 12090 consensus_queue.cc:237] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:24.970093 11603 catalog_manager.cc:5495] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b reported cstate change: term changed from 0 to 1, leader changed from <none> to 88d6b57e91a549d196c385819186d09b (127.29.159.66). New cstate: current_term: 1 leader_uuid: "88d6b57e91a549d196c385819186d09b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } health_report { overall_health: UNKNOWN } } }
I20260927 18:49:25.000306 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:24.996955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.29.159.66:33857)
0927 18:49:24.997084 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:24.997087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:24.997102 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:49:24.997234 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:24.997237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:24.997292 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:24.997396 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:49:24.997589 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.998079 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.998913 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:24.999118 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:24.999189 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:24.999557 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:49:24.999970 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:49:25.000065 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:25.000131 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 18:49:25.000179 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:49:25.006316 11753 raft_consensus.cc:1261] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Refusing update from remote peer 88d6b57e91a549d196c385819186d09b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:25.006316 12021 raft_consensus.cc:1261] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 1 FOLLOWER]: Refusing update from remote peer 88d6b57e91a549d196c385819186d09b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:25.007089 12090 consensus_queue.cc:1035] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:25.007400 12094 consensus_queue.cc:1035] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:25.017113 12098 mvcc.cc:205] Tried to move back new op lower bound from 7334031216658239488 to 7334031216505675776. Current Snapshot: MvccSnapshot[applied={T|T < 7334031216658239488}]
I20260927 18:49:25.017303 12099 mvcc.cc:205] Tried to move back new op lower bound from 7334031216658239488 to 7334031216505675776. Current Snapshot: MvccSnapshot[applied={T|T < 7334031216658239488}]
I20260927 18:49:25.024418 11608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:25.024605 11608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:25.026921 11608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Sending DeleteTablet for 3 replicas of tablet 96462d42d08546d5a1b1b1a4287259f8
I20260927 18:49:25.027665 12001 tablet_service.cc:1514] Processing DeleteTablet for tablet 96462d42d08546d5a1b1b1a4287259f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:25 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:34706
I20260927 18:49:25.027753 11733 tablet_service.cc:1514] Processing DeleteTablet for tablet 96462d42d08546d5a1b1b1a4287259f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:25 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:38564
I20260927 18:49:25.027742 11867 tablet_service.cc:1514] Processing DeleteTablet for tablet 96462d42d08546d5a1b1b1a4287259f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:25 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:40256
I20260927 18:49:25.028189 12107 tablet_replica.cc:330] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: stopping tablet replica
I20260927 18:49:25.028273 12109 tablet_replica.cc:330] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b: stopping tablet replica
I20260927 18:49:25.028398 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 11670
I20260927 18:49:25.028524 12108 tablet_replica.cc:330] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594: stopping tablet replica
I20260927 18:49:25.028590 12109 raft_consensus.cc:2229] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:49:25.028978 12109 raft_consensus.cc:2258] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:25.029165 12108 raft_consensus.cc:2229] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:25.029523 12108 raft_consensus.cc:2258] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:25.030834 12109 ts_tablet_manager.cc:1905] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:25.030992 12108 ts_tablet_manager.cc:1905] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:25.033587 12109 ts_tablet_manager.cc:1918] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:25.033689 12109 log.cc:1196] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/96462d42d08546d5a1b1b1a4287259f8
I20260927 18:49:25.034019 12109 ts_tablet_manager.cc:1939] T 96462d42d08546d5a1b1b1a4287259f8 P 88d6b57e91a549d196c385819186d09b: Deleting consensus metadata
I20260927 18:49:25.034466 12108 ts_tablet_manager.cc:1918] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:25.034614 12108 log.cc:1196] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/96462d42d08546d5a1b1b1a4287259f8
I20260927 18:49:25.034993 12108 ts_tablet_manager.cc:1939] T 96462d42d08546d5a1b1b1a4287259f8 P 88b6e3b4e1c94ffaa9b90561e127b594: Deleting consensus metadata
I20260927 18:49:25.035470 11594 catalog_manager.cc:4841] TS 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857): tablet 96462d42d08546d5a1b1b1a4287259f8 (table test-table [id=d680a339960e45a2a65d4e84803e671b]) successfully deleted
I20260927 18:49:25.036191 11594 catalog_manager.cc:4841] TS 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751): tablet 96462d42d08546d5a1b1b1a4287259f8 (table test-table [id=d680a339960e45a2a65d4e84803e671b]) successfully deleted
W20260927 18:49:25.037555 11594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:46481 (error 108)
I20260927 18:49:25.037825 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 11804
I20260927 18:49:25.038944 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.038642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:46481 (local address 127.0.0.1:38576)
0927 18:49:25.038784 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:25.038861 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 18:49:25.039178 11594 catalog_manager.cc:4584] TS 19bfd55a6c764979a211402015c5dacb (127.29.159.65:46481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96462d42d08546d5a1b1b1a4287259f8: Network error: Client connection negotiation failed: client connection to 127.29.159.65:46481: connect: Connection refused (error 111)
I20260927 18:49:25.043998 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 11938
I20260927 18:49:25.049103 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 11576
I20260927 18:49:25.055500 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41559
--webserver_interface=127.29.159.126
--webserver_port=40475
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:41559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:25.164124 12111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:25.164440 12111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:25.164522 12111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:25.168573 12111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:25.168664 12111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:25.168701 12111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:25.168761 12111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:25.168800 12111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:25.174140 12111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:41559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:25.175508 12111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:25.176528 12111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:25.189364 12118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:25.189375 12117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:25.189570 12111 server_base.cc:1034] running on GCE node
W20260927 18:49:25.189385 12120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:25.190383 12111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:25.191614 12111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:25.192803 12111 hybrid_clock.cc:648] HybridClock initialized: now 1790534965192782 us; error 31 us; skew 500 ppm
Sep 27 18:49:25 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:25.195652 12111 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:49:25.196493 12111 webserver.cc:460] Webserver started at http://127.29.159.126:40475/ using document root <none> and password file <none>
I20260927 18:49:25.197038 12111 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:25.197108 12111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:25.200898 12111 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:49:25.203332 12127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:25.204655 12111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:25.204761 12111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "d84c9a30434541899b41a22b20dfc13f"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "534f3ac80b64063cebe0c757902bfa37"
server_key_iv: "d3bd5be9b17fc7b65bacfa9e8d05a4fe"
server_key_version: "encryptionkey@0"
I20260927 18:49:25.205343 12111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:25.218792 12111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:25.221807 12111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:25.221969 12111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:25.229226 12111 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:41559
I20260927 18:49:25.229516 12179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:41559 every 8 connection(s)
I20260927 18:49:25.230410 12111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:25.231642 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 12111
I20260927 18:49:25.232049 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:46481
--local_ip_for_outbound_sockets=127.29.159.65
--tserver_master_addrs=127.29.159.126:41559
--webserver_port=32987
--webserver_interface=127.29.159.65
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:25.235606 12180 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:49:25.238262 12180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Bootstrap starting.
I20260927 18:49:25.261480 12180 log.cc:824] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:25.272779 12180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:49:25.273250 12180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Bootstrap complete.
I20260927 18:49:25.279161 12180 raft_consensus.cc:348] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:25.279925 12180 raft_consensus.cc:729] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84c9a30434541899b41a22b20dfc13f, State: Initialized, Role: FOLLOWER
I20260927 18:49:25.280534 12180 consensus_queue.cc:260] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:25.280700 12180 raft_consensus.cc:388] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:25.280778 12180 raft_consensus.cc:482] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:25.280893 12180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:49:25.283602 12180 raft_consensus.cc:504] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:25.284018 12180 leader_election.cc:302] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d84c9a30434541899b41a22b20dfc13f; no voters:
I20260927 18:49:25.284392 12180 leader_election.cc:288] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:49:25.284541 12184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:49:25.284845 12184 raft_consensus.cc:686] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 2 LEADER]: Becoming Leader. State: Replica: d84c9a30434541899b41a22b20dfc13f, State: Running, Role: LEADER
I20260927 18:49:25.285184 12184 consensus_queue.cc:237] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:25.285706 12180 sys_catalog.cc:564] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:25.286837 12185 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d84c9a30434541899b41a22b20dfc13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } } }
I20260927 18:49:25.286899 12186 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d84c9a30434541899b41a22b20dfc13f. Latest consensus state: current_term: 2 leader_uuid: "d84c9a30434541899b41a22b20dfc13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } } }
I20260927 18:49:25.286966 12185 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:25.286966 12186 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:25.287374 12193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:25.292261 12193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:25.293128 12193 catalog_manager.cc:1259] Loaded cluster ID: c623aa9f592249b1b987dfdee4794ba8
I20260927 18:49:25.293201 12193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:25.294284 12193 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 18:49:25.297137 12193 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 18:49:25.301929 12193 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:25.302564 12193 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Loaded TSK: 0
I20260927 18:49:25.303565 12193 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:49:25.370296 12182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:25.370540 12182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:25.370617 12182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:25.374750 12182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:25.374817 12182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:25.374912 12182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:49:25.380174 12182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:46481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=32987
--tserver_master_addrs=127.29.159.126:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:25.381291 12182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:25.384289 12182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:25.395750 12207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:25.395852 12208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:25.395852 12210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:25.397147 12182 server_base.cc:1034] running on GCE node
I20260927 18:49:25.397755 12182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:25.398463 12182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:25.399633 12182 hybrid_clock.cc:648] HybridClock initialized: now 1790534965399608 us; error 44 us; skew 500 ppm
Sep 27 18:49:25 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:25.402616 12182 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:49:25.403680 12182 webserver.cc:460] Webserver started at http://127.29.159.65:32987/ using document root <none> and password file <none>
I20260927 18:49:25.404251 12182 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:25.404358 12182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:25.408495 12182 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:49:25.410876 12217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:25.412217 12182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:49:25.412350 12182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "19bfd55a6c764979a211402015c5dacb"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "1380b6db95c0a0ded47ae7d6aa3b6d78"
server_key_iv: "6aecb2743ce48fb12f0c1f9e0f70599a"
server_key_version: "encryptionkey@0"
I20260927 18:49:25.412914 12182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:25.424554 12182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:25.427922 12182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:25.428118 12182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:25.428732 12182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:25.429910 12224 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:49:25.434003 12182 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:49:25.434067 12182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 18:49:25.434131 12182 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:49:25.435804 12182 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:49:25.435859 12182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 18:49:25.436120 12224 tablet_bootstrap.cc:492] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Bootstrap starting.
I20260927 18:49:25.446679 12182 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:46481
I20260927 18:49:25.446686 12331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:46481 every 8 connection(s)
I20260927 18:49:25.447901 12182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:25.449509 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 12182
I20260927 18:49:25.449995 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:33857
--local_ip_for_outbound_sockets=127.29.159.66
--tserver_master_addrs=127.29.159.126:41559
--webserver_port=39223
--webserver_interface=127.29.159.66
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:25.458101 12224 log.cc:824] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:49:25 dist-test-slave-8tg2 krb5kdc[11563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:25.466125 12224 tablet_bootstrap.cc:492] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:49:25.466580 12224 tablet_bootstrap.cc:492] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Bootstrap complete.
I20260927 18:49:25.467916 12224 ts_tablet_manager.cc:1397] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.012s
I20260927 18:49:25.469138 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.450165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:41721)
0927 18:49:25.451788 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:25.451844 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:25.453034 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:25.454315 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:25.454328 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:25.454839 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:25.455575 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.455609 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.457521 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.457528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.459134 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.459149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.460148 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.460156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.460414 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.461165 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:25.461191 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:25.463207 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:25.465698 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:25.465704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:25.465718 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:25.466085 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:25.466494 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:25.466497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:25.466499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:25.466664 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:25.467336 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:25.467342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:25.467648 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 18:49:25.467855 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 18:49:25.468176 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"thread_start_us":154,"threads_started":1}
I20260927 18:49:25.469149 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.450156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:41721 (local address 127.29.159.126:41559)
0927 18:49:25.451505 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:25.451522 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:25.451987 (+ 465us) server_negotiation.cc:407] Connection header received
0927 18:49:25.453517 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:25.453529 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:25.453860 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:25.454136 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:25.455817 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.457323 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.459281 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.459990 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.460156 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.463472 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:25.463479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:25.463496 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:25.463530 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:25.465569 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:25.466210 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:25.466216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:25.466222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:25.466385 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:25.466753 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:25.466756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:25.466758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:25.467217 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:25.467352 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:25.468040 (+ 688us) server_negotiation.cc:299] Negotiation successful
0927 18:49:25.468347 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":98,"threads_started":1}
I20260927 18:49:25.471968 12332 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41559
I20260927 18:49:25.472188 12332 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:25.471843 12224 raft_consensus.cc:348] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:25.472369 12224 raft_consensus.cc:729] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bfd55a6c764979a211402015c5dacb, State: Initialized, Role: FOLLOWER
I20260927 18:49:25.472728 12332 heartbeater.cc:511] Master 127.29.159.126:41559 requested a full tablet report, sending...
I20260927 18:49:25.472842 12224 consensus_queue.cc:260] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:25.474504 12224 ts_tablet_manager.cc:1428] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:49:25.475564 12144 ts_manager.cc:194] Registered new tserver with Master: 19bfd55a6c764979a211402015c5dacb (127.29.159.65:46481)
I20260927 18:49:25.479514 12144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:41721
I20260927 18:49:25.481849 12332 heartbeater.cc:503] Master 127.29.159.126:41559 was elected leader, sending a full tablet report...
Sep 27 18:49:25 dist-test-slave-8tg2 krb5kdc[11563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:49:25.489826 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.478854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:46481 (local address 127.0.0.1:38592)
0927 18:49:25.479588 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:25.479659 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:25.479820 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:25.480369 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:25.480386 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:25.480822 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:25.480949 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.480957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.482358 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.482361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.483005 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.483014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.483834 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.483838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.483948 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.484519 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:25.484530 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:25.486110 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:25.488050 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:25.488060 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:25.488065 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:25.488286 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:25.488599 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:25.488602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:25.488604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:25.488688 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:25.489127 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:25.489133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:25.489264 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 18:49:25.489426 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 18:49:25.489646 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":110,"threads_started":1}
I20260927 18:49:25.489826 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.478889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.29.159.65:46481)
0927 18:49:25.479258 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:25.479269 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:25.479762 (+ 493us) server_negotiation.cc:407] Connection header received
0927 18:49:25.479964 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:25.479971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:25.480128 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:25.480266 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:25.481081 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.482243 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.483128 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.483709 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.483834 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.486257 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:25.486265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:25.486277 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:25.486308 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:25.487940 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:25.488407 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:25.488413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:25.488415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:25.488498 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:25.488783 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:25.488786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:25.488788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:25.489013 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:25.489153 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:25.489513 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 18:49:25.489662 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":97,"threads_started":1}
I20260927 18:49:25.490816 12266 tablet_service.cc:1514] Processing DeleteTablet for tablet 96462d42d08546d5a1b1b1a4287259f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:25 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:38592
I20260927 18:49:25.491783 12341 tablet_replica.cc:330] stopping tablet replica
I20260927 18:49:25.492059 12341 raft_consensus.cc:2229] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:25.492292 12341 raft_consensus.cc:2258] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:25.493186 12341 ts_tablet_manager.cc:1905] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:25.496842 12341 ts_tablet_manager.cc:1918] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:25.496986 12341 log.cc:1196] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal/wals/96462d42d08546d5a1b1b1a4287259f8
I20260927 18:49:25.497270 12341 ts_tablet_manager.cc:1939] T 96462d42d08546d5a1b1b1a4287259f8 P 19bfd55a6c764979a211402015c5dacb: Deleting consensus metadata
I20260927 18:49:25.498929 12130 catalog_manager.cc:4841] TS 19bfd55a6c764979a211402015c5dacb (127.29.159.65:46481): tablet 96462d42d08546d5a1b1b1a4287259f8 (table test-table [id=d680a339960e45a2a65d4e84803e671b]) successfully deleted
W20260927 18:49:25.570654 12336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:25.570919 12336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:25.570960 12336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:25.575013 12336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:25.575068 12336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:25.575125 12336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:49:25.580299 12336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:33857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=39223
--tserver_master_addrs=127.29.159.126:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:25.581210 12336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:25.582144 12336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:25.591826 12346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:25.591831 12347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:25.591806 12349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:25.592092 12336 server_base.cc:1034] running on GCE node
I20260927 18:49:25.592528 12336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:25.593154 12336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:25.594342 12336 hybrid_clock.cc:648] HybridClock initialized: now 1790534965594331 us; error 38 us; skew 500 ppm
Sep 27 18:49:25 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:25.597162 12336 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:49:25.598107 12336 webserver.cc:460] Webserver started at http://127.29.159.66:39223/ using document root <none> and password file <none>
I20260927 18:49:25.598637 12336 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:25.598711 12336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:25.602430 12336 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:49:25.604437 12356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:25.605659 12336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:49:25.605880 12336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "88d6b57e91a549d196c385819186d09b"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "42b9be8a1814e03539e585cc728a2e76"
server_key_iv: "b8c434d16e111f2b82844b5c216b7717"
server_key_version: "encryptionkey@0"
I20260927 18:49:25.606393 12336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:25.623334 12336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:25.626848 12336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:25.627053 12336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:25.627729 12336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:25.628562 12336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:25.628609 12336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:25.628676 12336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:25.628717 12336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:25.637554 12336 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:33857
I20260927 18:49:25.637579 12469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:33857 every 8 connection(s)
I20260927 18:49:25.638577 12336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
Sep 27 18:49:25 dist-test-slave-8tg2 krb5kdc[11563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:25.648669 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 12336
I20260927 18:49:25.649181 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:35751
--local_ip_for_outbound_sockets=127.29.159.67
--tserver_master_addrs=127.29.159.126:41559
--webserver_port=34517
--webserver_interface=127.29.159.67
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:25.652426 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.640543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:51513 (local address 127.29.159.126:41559)
0927 18:49:25.640697 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:25.640701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:25.641703 (+ 1002us) server_negotiation.cc:407] Connection header received
0927 18:49:25.642612 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:25.642616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:25.642681 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:25.642786 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:25.643977 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.644503 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.645617 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.645860 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.645974 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.648460 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:25.648463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:25.648465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:25.648490 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:25.649995 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:25.650650 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:25.650657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:25.650660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:25.650757 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:25.651083 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:25.651088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:25.651090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:25.651333 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:25.651468 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:25.652000 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 18:49:25.652157 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:49:25.653059 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.640910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:51513)
0927 18:49:25.641553 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:25.641591 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:25.642392 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:25.642970 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:25.642977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:25.643318 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:25.643796 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.643808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.644631 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.644635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.645488 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.645497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.646006 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.646010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.646179 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.646713 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:25.646728 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:25.648220 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:25.650113 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:25.650123 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:25.650141 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:25.650533 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:25.650856 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:25.650858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:25.650860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:25.650993 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:25.651498 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:25.651503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:25.651734 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 18:49:25.651922 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 18:49:25.652175 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":114,"threads_started":1}
I20260927 18:49:25.654337 12470 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41559
I20260927 18:49:25.654598 12470 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:25.655117 12470 heartbeater.cc:511] Master 127.29.159.126:41559 requested a full tablet report, sending...
I20260927 18:49:25.656332 12144 ts_manager.cc:194] Registered new tserver with Master: 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857)
I20260927 18:49:25.656956 12144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:51513
W20260927 18:49:25.762591 12473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:25.762917 12473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:25.763007 12473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:25.767223 12473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:25.767310 12473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:25.767406 12473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:49:25.772810 12473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:35751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=34517
--tserver_master_addrs=127.29.159.126:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:25.773859 12473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:25.775168 12473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:25.784889 12478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:25.784858 12481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:25.785161 12473 server_base.cc:1034] running on GCE node
W20260927 18:49:25.784951 12479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:25.785621 12473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:25.786435 12473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:25.787580 12473 hybrid_clock.cc:648] HybridClock initialized: now 1790534965787563 us; error 30 us; skew 500 ppm
Sep 27 18:49:25 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:25.790263 12473 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:49:25.791296 12473 webserver.cc:460] Webserver started at http://127.29.159.67:34517/ using document root <none> and password file <none>
I20260927 18:49:25.791924 12473 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:25.792019 12473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:25.795897 12473 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 18:49:25.797932 12488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:25.799207 12473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:49:25.799346 12473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "88b6e3b4e1c94ffaa9b90561e127b594"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "f8920269a0470bf5d3faa7ad8c4ff6c8"
server_key_iv: "8ef0caf9e2f071e1695f4bb6a60def6d"
server_key_version: "encryptionkey@0"
I20260927 18:49:25.799940 12473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:25.814394 12473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:25.817412 12473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:25.817610 12473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:25.818163 12473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:25.818959 12473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:25.819028 12473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:25.819095 12473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:25.819142 12473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:25.828269 12473 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:35751
I20260927 18:49:25.828295 12601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:35751 every 8 connection(s)
I20260927 18:49:25.829245 12473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:25.836197 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 12473
Sep 27 18:49:25 dist-test-slave-8tg2 krb5kdc[11563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:25.843153 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.830971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:48451 (local address 127.29.159.126:41559)
0927 18:49:25.831129 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:25.831133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:25.832232 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 18:49:25.833514 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:25.833516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:25.833560 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:25.833631 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:25.834763 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.835254 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.836498 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.836821 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.836950 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.839273 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:25.839277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:25.839279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:25.839306 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:25.840853 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:25.841461 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:25.841463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:25.841465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:25.841536 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:25.841905 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:25.841908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:25.841910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:25.842140 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:25.842263 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:25.842812 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 18:49:25.842948 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:49:25.844005 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.831387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:48451)
0927 18:49:25.832093 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:25.832132 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:25.833343 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:25.833807 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:25.833813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:25.834150 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:25.834644 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.834657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.835354 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.835356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.836385 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.836394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.836983 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.836987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.837155 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.837695 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:25.837715 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:25.839117 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:25.840976 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:25.840982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:25.840996 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:25.841361 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:25.841629 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:25.841633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:25.841635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:25.841813 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:25.842285 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:25.842293 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:25.842572 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 18:49:25.842766 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 18:49:25.843050 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":109,"threads_started":1}
I20260927 18:49:25.845239 12602 heartbeater.cc:347] Connected to a master server at 127.29.159.126:41559
I20260927 18:49:25.845403 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.836917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.29.159.126:41559)
0927 18:49:25.837147 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:25.837150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:25.837418 (+ 268us) server_negotiation.cc:407] Connection header received
0927 18:49:25.837552 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:25.837557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:25.837602 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:25.837689 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:25.838424 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.838982 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.840033 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.840277 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.840406 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.841929 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:25.841932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:25.841938 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:25.841961 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:25.843612 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:25.844107 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:25.844111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:25.844113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:25.844211 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:25.844484 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:25.844487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:25.844489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:25.844671 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:25.844890 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:25.845126 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 18:49:25.845249 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":57,"threads_started":1}
I20260927 18:49:25.845479 12602 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:25.845986 12602 heartbeater.cc:511] Master 127.29.159.126:41559 requested a full tablet report, sending...
I20260927 18:49:25.847079 12143 ts_manager.cc:194] Registered new tserver with Master: 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751)
I20260927 18:49:25.847733 12143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:48451
I20260927 18:49:25.858686 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:49:25.863066 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.859355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.29.159.126:41559)
0927 18:49:25.859507 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:25.859511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:25.859701 (+ 190us) server_negotiation.cc:407] Connection header received
0927 18:49:25.859881 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:25.859884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:25.859967 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:25.860075 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:49:25.860307 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.860796 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.861706 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.861939 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.862055 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.862349 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:49:25.862688 (+ 339us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:49:25.862817 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:25.862884 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 18:49:25.862936 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:49:25.863888 12143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:49:25.866997 12143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:49:25.874897 12266 tablet_service.cc:1467] Processing CreateTablet for tablet 3def9ff332d04542b2f5c16fa10127a2 (DEFAULT_TABLE table=test-table [id=609583133ca042d4894fc1742ace17ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:25.875568 12266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3def9ff332d04542b2f5c16fa10127a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:25.878310 12224 tablet_bootstrap.cc:492] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: Bootstrap starting.
I20260927 18:49:25.879002 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.873506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:33857 (local address 127.0.0.1:40262)
0927 18:49:25.873613 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:25.873629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:25.873710 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:25.875063 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:25.875066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:25.875088 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:25.875185 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.875194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.876301 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.876307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.877435 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.877445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.878576 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.878581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.878675 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.878713 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:49:25.878783 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 18:49:25.878858 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":41}
I20260927 18:49:25.879128 12224 tablet_bootstrap.cc:654] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:25.879323 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.873937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.29.159.67:35751)
0927 18:49:25.874251 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:25.874262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:25.874284 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:49:25.874353 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:25.874357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:25.874568 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:25.874717 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:25.875143 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.876195 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.877643 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.878518 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.878609 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.878751 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:25.879051 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 18:49:25.879160 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260927 18:49:25.879572 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.873806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:35751 (local address 127.0.0.1:34718)
0927 18:49:25.874040 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:25.874071 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:25.874161 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:25.874902 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:25.874905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:25.874926 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:25.875015 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.875021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.876341 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.876345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.877538 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:25.877548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.878809 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.878812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:25.878890 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.878910 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:49:25.878982 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:49:25.879050 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":48,"thread_start_us":98,"threads_started":1}
I20260927 18:49:25.880333 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:25.874380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.29.159.66:33857)
0927 18:49:25.874692 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:25.874703 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:25.874731 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:49:25.874797 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:25.874800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:25.874934 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:25.875144 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:25.875304 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.876164 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.877562 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:25.878431 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:25.879928 (+ 1497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:25.880070 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:25.880116 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 18:49:25.880199 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":126,"threads_started":1}
I20260927 18:49:25.880522 12224 tablet_bootstrap.cc:492] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: No bootstrap required, opened a new log
I20260927 18:49:25.880606 12224 ts_tablet_manager.cc:1397] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:49:25.881019 12224 raft_consensus.cc:348] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:25.881135 12224 raft_consensus.cc:374] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:25.881172 12224 raft_consensus.cc:729] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bfd55a6c764979a211402015c5dacb, State: Initialized, Role: FOLLOWER
I20260927 18:49:25.881295 12224 consensus_queue.cc:260] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:25.881542 12224 ts_tablet_manager.cc:1428] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 18:49:25.882874 12536 tablet_service.cc:1467] Processing CreateTablet for tablet 3def9ff332d04542b2f5c16fa10127a2 (DEFAULT_TABLE table=test-table [id=609583133ca042d4894fc1742ace17ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:25.882893 12404 tablet_service.cc:1467] Processing CreateTablet for tablet 3def9ff332d04542b2f5c16fa10127a2 (DEFAULT_TABLE table=test-table [id=609583133ca042d4894fc1742ace17ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:25.883893 12404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3def9ff332d04542b2f5c16fa10127a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:25.883972 12536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3def9ff332d04542b2f5c16fa10127a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:25.890266 12615 tablet_bootstrap.cc:492] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b: Bootstrap starting.
I20260927 18:49:25.890275 12616 tablet_bootstrap.cc:492] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594: Bootstrap starting.
I20260927 18:49:25.892669 12615 tablet_bootstrap.cc:654] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:25.892692 12616 tablet_bootstrap.cc:654] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:25.893376 12616 log.cc:824] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:25.893376 12615 log.cc:824] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:25.895104 12615 tablet_bootstrap.cc:492] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b: No bootstrap required, opened a new log
I20260927 18:49:25.895308 12615 ts_tablet_manager.cc:1397] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:49:25.896826 12616 tablet_bootstrap.cc:492] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594: No bootstrap required, opened a new log
I20260927 18:49:25.897010 12616 ts_tablet_manager.cc:1397] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 18:49:25.898447 12615 raft_consensus.cc:348] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:25.898693 12615 raft_consensus.cc:374] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:25.898778 12615 raft_consensus.cc:729] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d6b57e91a549d196c385819186d09b, State: Initialized, Role: FOLLOWER
I20260927 18:49:25.899163 12615 consensus_queue.cc:260] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:25.899844 12470 heartbeater.cc:503] Master 127.29.159.126:41559 was elected leader, sending a full tablet report...
I20260927 18:49:25.900120 12615 ts_tablet_manager.cc:1428] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 18:49:25.900120 12616 raft_consensus.cc:348] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:25.900374 12616 raft_consensus.cc:374] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:25.900439 12616 raft_consensus.cc:729] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b6e3b4e1c94ffaa9b90561e127b594, State: Initialized, Role: FOLLOWER
I20260927 18:49:25.900835 12616 consensus_queue.cc:260] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:25.901454 12602 heartbeater.cc:503] Master 127.29.159.126:41559 was elected leader, sending a full tablet report...
I20260927 18:49:25.901840 12616 ts_tablet_manager.cc:1428] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 18:49:25.949287 12333 tablet.cc:2369] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:26.000056 12338 raft_consensus.cc:482] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:26.000289 12338 raft_consensus.cc:504] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:26.001423 12338 leader_election.cc:288] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857), 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751)
I20260927 18:49:26.005610 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.001828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:53859 (local address 127.29.159.66:33857)
0927 18:49:26.001969 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:26.001975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:26.002075 (+ 100us) server_negotiation.cc:407] Connection header received
0927 18:49:26.002269 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:26.002273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:26.002335 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:26.002413 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:26.002730 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:26.003402 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:26.004615 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:26.005017 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:26.005145 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:26.005195 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:26.005413 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 18:49:26.005481 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:49:26.005635 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.001666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:33857 (local address 127.29.159.65:53859)
0927 18:49:26.001968 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.001984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:26.002112 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:26.002447 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:26.002450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:26.002476 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:26.002616 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:26.002624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:26.003527 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:26.003532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:26.004501 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:26.004510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:26.005136 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:26.005140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:26.005271 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:26.005307 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:49:26.005395 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:49:26.005466 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":31,"thread_start_us":86,"threads_started":1}
I20260927 18:49:26.005649 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.002210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:56319 (local address 127.29.159.67:35751)
0927 18:49:26.002342 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:26.002346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:26.002522 (+ 176us) server_negotiation.cc:407] Connection header received
0927 18:49:26.002649 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:26.002652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:26.002704 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:26.002793 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:26.003009 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:26.003581 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:26.004665 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:26.005073 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:26.005197 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:26.005251 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:26.005464 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 18:49:26.005528 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:49:26.005663 12622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.002092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:35751 (local address 127.29.159.65:56319)
0927 18:49:26.002449 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.002461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:26.002543 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:26.002802 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:26.002805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:26.002822 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:26.002907 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:26.002913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:26.003689 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:26.003693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:26.004537 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:26.004547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:26.005195 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:26.005199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:26.005343 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:26.005361 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:49:26.005449 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:49:26.005518 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":24,"thread_start_us":114,"threads_started":1}
I20260927 18:49:26.006477 12556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3def9ff332d04542b2f5c16fa10127a2" candidate_uuid: "19bfd55a6c764979a211402015c5dacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" is_pre_election: true
I20260927 18:49:26.006474 12424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3def9ff332d04542b2f5c16fa10127a2" candidate_uuid: "19bfd55a6c764979a211402015c5dacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88d6b57e91a549d196c385819186d09b" is_pre_election: true
I20260927 18:49:26.006786 12556 raft_consensus.cc:2454] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bfd55a6c764979a211402015c5dacb in term 0.
I20260927 18:49:26.006786 12424 raft_consensus.cc:2454] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bfd55a6c764979a211402015c5dacb in term 0.
I20260927 18:49:26.007210 12220 leader_election.cc:302] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bfd55a6c764979a211402015c5dacb, 88b6e3b4e1c94ffaa9b90561e127b594; no voters:
I20260927 18:49:26.007442 12338 raft_consensus.cc:2781] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:49:26.007550 12338 raft_consensus.cc:482] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:49:26.007608 12338 raft_consensus.cc:3037] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:26.008772 12338 raft_consensus.cc:504] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:26.009104 12338 leader_election.cc:288] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 1 election: Requested vote from peers 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857), 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751)
I20260927 18:49:26.009513 12424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3def9ff332d04542b2f5c16fa10127a2" candidate_uuid: "19bfd55a6c764979a211402015c5dacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88d6b57e91a549d196c385819186d09b"
I20260927 18:49:26.009557 12556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3def9ff332d04542b2f5c16fa10127a2" candidate_uuid: "19bfd55a6c764979a211402015c5dacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b6e3b4e1c94ffaa9b90561e127b594"
I20260927 18:49:26.009651 12424 raft_consensus.cc:3037] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:26.009657 12556 raft_consensus.cc:3037] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:26.010941 12424 raft_consensus.cc:2454] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bfd55a6c764979a211402015c5dacb in term 1.
I20260927 18:49:26.010941 12556 raft_consensus.cc:2454] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bfd55a6c764979a211402015c5dacb in term 1.
I20260927 18:49:26.011191 12220 leader_election.cc:302] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bfd55a6c764979a211402015c5dacb, 88b6e3b4e1c94ffaa9b90561e127b594; no voters:
I20260927 18:49:26.011349 12338 raft_consensus.cc:2781] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:26.011529 12338 raft_consensus.cc:686] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 LEADER]: Becoming Leader. State: Replica: 19bfd55a6c764979a211402015c5dacb, State: Running, Role: LEADER
I20260927 18:49:26.011813 12338 consensus_queue.cc:237] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:26.014681 12144 catalog_manager.cc:5495] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb reported cstate change: term changed from 0 to 1, leader changed from <none> to 19bfd55a6c764979a211402015c5dacb (127.29.159.65). New cstate: current_term: 1 leader_uuid: "19bfd55a6c764979a211402015c5dacb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } health_report { overall_health: HEALTHY } } }
I20260927 18:49:26.038446 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.034737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.29.159.65:46481)
0927 18:49:26.035059 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:26.035066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:26.035077 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:49:26.035139 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:26.035142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:26.035191 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:26.035321 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:49:26.035592 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:26.036122 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:26.036965 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:26.037204 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:26.037314 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:26.037607 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:49:26.038028 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:49:26.038156 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:26.038228 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 18:49:26.038283 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":117,"threads_started":1}
I20260927 18:49:26.042502 12556 raft_consensus.cc:1261] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 1 FOLLOWER]: Refusing update from remote peer 19bfd55a6c764979a211402015c5dacb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:26.042518 12424 raft_consensus.cc:1261] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b [term 1 FOLLOWER]: Refusing update from remote peer 19bfd55a6c764979a211402015c5dacb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:26.043264 12338 consensus_queue.cc:1035] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [LEADER]: Connected to new peer: Peer: permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:26.043529 12623 consensus_queue.cc:1035] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:26.051764 12629 mvcc.cc:205] Tried to move back new op lower bound from 7334031220903280640 to 7334031220787253248. Current Snapshot: MvccSnapshot[applied={T|T < 7334031220903280640}]
I20260927 18:49:26.058969 12144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:26.059158 12144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:26.061616 12144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Sending DeleteTablet for 3 replicas of tablet 3def9ff332d04542b2f5c16fa10127a2
I20260927 18:49:26.062264 12404 tablet_service.cc:1514] Processing DeleteTablet for tablet 3def9ff332d04542b2f5c16fa10127a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:26 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:40262
I20260927 18:49:26.062371 12536 tablet_service.cc:1514] Processing DeleteTablet for tablet 3def9ff332d04542b2f5c16fa10127a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:26 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:34718
I20260927 18:49:26.062638 12266 tablet_service.cc:1514] Processing DeleteTablet for tablet 3def9ff332d04542b2f5c16fa10127a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:26 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:38592
I20260927 18:49:26.062768 12636 tablet_replica.cc:330] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b: stopping tablet replica
I20260927 18:49:26.063086 12636 raft_consensus.cc:2229] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:26.063131 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 12182
I20260927 18:49:26.063944 12637 tablet_replica.cc:330] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594: stopping tablet replica
I20260927 18:49:26.063398 12636 raft_consensus.cc:2258] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:26.064244 12637 raft_consensus.cc:2229] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:26.064524 12637 raft_consensus.cc:2258] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:26.065719 12636 ts_tablet_manager.cc:1905] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:26.065994 12637 ts_tablet_manager.cc:1905] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:26.068156 12637 ts_tablet_manager.cc:1918] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:26.068279 12637 log.cc:1196] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/3def9ff332d04542b2f5c16fa10127a2
I20260927 18:49:26.068594 12637 ts_tablet_manager.cc:1939] T 3def9ff332d04542b2f5c16fa10127a2 P 88b6e3b4e1c94ffaa9b90561e127b594: Deleting consensus metadata
I20260927 18:49:26.069079 12636 ts_tablet_manager.cc:1918] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:26.069186 12636 log.cc:1196] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/3def9ff332d04542b2f5c16fa10127a2
I20260927 18:49:26.069514 12636 ts_tablet_manager.cc:1939] T 3def9ff332d04542b2f5c16fa10127a2 P 88d6b57e91a549d196c385819186d09b: Deleting consensus metadata
I20260927 18:49:26.069825 12130 catalog_manager.cc:4841] TS 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751): tablet 3def9ff332d04542b2f5c16fa10127a2 (table test-table [id=609583133ca042d4894fc1742ace17ca]) successfully deleted
I20260927 18:49:26.070914 12130 catalog_manager.cc:4841] TS 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857): tablet 3def9ff332d04542b2f5c16fa10127a2 (table test-table [id=609583133ca042d4894fc1742ace17ca]) successfully deleted
W20260927 18:49:26.071163 12130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:46481 (error 108)
I20260927 18:49:26.071251 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 12336
I20260927 18:49:26.072844 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.072534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:46481 (local address 127.0.0.1:38610)
0927 18:49:26.072678 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.072775 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 18:49:26.072989 12130 catalog_manager.cc:4584] TS 19bfd55a6c764979a211402015c5dacb (127.29.159.65:46481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3def9ff332d04542b2f5c16fa10127a2: Network error: Client connection negotiation failed: client connection to 127.29.159.65:46481: connect: Connection refused (error 111)
I20260927 18:49:26.077982 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 12473
I20260927 18:49:26.083201 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 12111
I20260927 18:49:26.089390 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41559
--webserver_interface=127.29.159.126
--webserver_port=40475
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:41559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:26.202421 12640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:26.202757 12640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:26.202836 12640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:26.206875 12640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:26.206965 12640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:26.207003 12640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:26.207041 12640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:26.207080 12640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:26.212404 12640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:41559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:41559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:26.213446 12640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:26.214493 12640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:26.223256 12645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:26.223361 12640 server_base.cc:1034] running on GCE node
W20260927 18:49:26.223256 12646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:26.223260 12648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:26.223964 12640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:26.225083 12640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:26.226276 12640 hybrid_clock.cc:648] HybridClock initialized: now 1790534966226228 us; error 61 us; skew 500 ppm
Sep 27 18:49:26 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534966, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:26.229053 12640 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:49:26.229993 12640 webserver.cc:460] Webserver started at http://127.29.159.126:40475/ using document root <none> and password file <none>
I20260927 18:49:26.230553 12640 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:26.230644 12640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:26.234419 12640 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:49:26.236603 12655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:26.237852 12640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:49:26.237984 12640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "d84c9a30434541899b41a22b20dfc13f"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "534f3ac80b64063cebe0c757902bfa37"
server_key_iv: "d3bd5be9b17fc7b65bacfa9e8d05a4fe"
server_key_version: "encryptionkey@0"
I20260927 18:49:26.238543 12640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:26.250546 12640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:26.253786 12640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:26.253983 12640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:26.261166 12640 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:41559
I20260927 18:49:26.261456 12707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:41559 every 8 connection(s)
I20260927 18:49:26.262247 12640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:26.265745 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 12640
I20260927 18:49:26.266227 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:46481
--local_ip_for_outbound_sockets=127.29.159.65
--tserver_master_addrs=127.29.159.126:41559
--webserver_port=32987
--webserver_interface=127.29.159.65
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:26.266939 12708 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:49:26.269063 12708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Bootstrap starting.
I20260927 18:49:26.293653 12708 log.cc:824] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:26.307698 12708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:49:26.308149 12708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f: Bootstrap complete.
I20260927 18:49:26.313632 12708 raft_consensus.cc:348] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:26.314148 12708 raft_consensus.cc:729] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84c9a30434541899b41a22b20dfc13f, State: Initialized, Role: FOLLOWER
I20260927 18:49:26.314680 12708 consensus_queue.cc:260] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:26.314831 12708 raft_consensus.cc:388] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:26.314888 12708 raft_consensus.cc:482] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:26.314975 12708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:49:26.317715 12708 raft_consensus.cc:504] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:26.318089 12708 leader_election.cc:302] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d84c9a30434541899b41a22b20dfc13f; no voters:
I20260927 18:49:26.318460 12708 leader_election.cc:288] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:49:26.318635 12712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:49:26.318926 12712 raft_consensus.cc:686] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [term 3 LEADER]: Becoming Leader. State: Replica: d84c9a30434541899b41a22b20dfc13f, State: Running, Role: LEADER
I20260927 18:49:26.319252 12712 consensus_queue.cc:237] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } }
I20260927 18:49:26.319720 12708 sys_catalog.cc:564] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:26.320724 12713 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d84c9a30434541899b41a22b20dfc13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } } }
I20260927 18:49:26.320854 12713 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:26.321185 12714 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d84c9a30434541899b41a22b20dfc13f. Latest consensus state: current_term: 3 leader_uuid: "d84c9a30434541899b41a22b20dfc13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84c9a30434541899b41a22b20dfc13f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 41559 } } }
I20260927 18:49:26.321280 12714 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84c9a30434541899b41a22b20dfc13f [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:26.321725 12721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:26.327555 12721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:26.328289 12721 catalog_manager.cc:1259] Loaded cluster ID: c623aa9f592249b1b987dfdee4794ba8
I20260927 18:49:26.328356 12721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 18:49:26.329468 12721 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdc82cc19ad google::LogMessage::Fail() at ??:0
@ 0x7fdc82cc488a google::LogMessage::SendToLog() at ??:0
@ 0x7fdc82cc14de google::LogMessage::Flush() at ??:0
@ 0x7fdc82cc2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdc8d600fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fdc8d601d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdc8d5fe579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fdc8d642640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdc84f133d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdc84f13c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fdc84f153f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdc84f04512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdc865486db start_thread at ??:0
@ 0x7fdc817bf61f clone at ??:0
W20260927 18:49:26.416221 12710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:26.416527 12710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:26.416604 12710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:26.421732 12710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:26.421839 12710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:26.421940 12710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:49:26.428025 12710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:46481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=32987
--tserver_master_addrs=127.29.159.126:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:26.429214 12710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:26.430328 12710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:26.440857 12750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:26.440850 12749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:26.440830 12752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:26.441407 12710 server_base.cc:1034] running on GCE node
I20260927 18:49:26.441813 12710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:26.442517 12710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:26.443725 12710 hybrid_clock.cc:648] HybridClock initialized: now 1790534966443689 us; error 47 us; skew 500 ppm
Sep 27 18:49:26 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534966, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:26.446627 12710 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:49:26.447626 12710 webserver.cc:460] Webserver started at http://127.29.159.65:32987/ using document root <none> and password file <none>
I20260927 18:49:26.448169 12710 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:26.448256 12710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:26.452026 12710 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 18:49:26.454203 12759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:26.455575 12710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:26.455703 12710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "19bfd55a6c764979a211402015c5dacb"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "1380b6db95c0a0ded47ae7d6aa3b6d78"
server_key_iv: "6aecb2743ce48fb12f0c1f9e0f70599a"
server_key_version: "encryptionkey@0"
I20260927 18:49:26.456255 12710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:26.476402 12710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:26.479869 12710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:26.480075 12710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:26.480669 12710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:26.481837 12766 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:49:26.487671 12710 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:49:26.487746 12710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 18:49:26.487848 12710 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:49:26.489466 12710 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:49:26.489534 12710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 18:49:26.489817 12766 tablet_bootstrap.cc:492] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: Bootstrap starting.
I20260927 18:49:26.500085 12710 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:46481
I20260927 18:49:26.500174 12873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:46481 every 8 connection(s)
I20260927 18:49:26.501039 12710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:26.504614 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 12710
I20260927 18:49:26.505020 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:33857
--local_ip_for_outbound_sockets=127.29.159.66
--tserver_master_addrs=127.29.159.126:41559
--webserver_port=39223
--webserver_interface=127.29.159.66
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:26.514910 12766 log.cc:824] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:26.522199 12766 tablet_bootstrap.cc:492] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:49:26.522585 12766 tablet_bootstrap.cc:492] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: Bootstrap complete.
I20260927 18:49:26.523820 12766 ts_tablet_manager.cc:1397] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.011s
I20260927 18:49:26.526218 12766 raft_consensus.cc:348] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:26.526657 12766 raft_consensus.cc:729] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bfd55a6c764979a211402015c5dacb, State: Initialized, Role: FOLLOWER
I20260927 18:49:26.527153 12766 consensus_queue.cc:260] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:26.528162 12766 ts_tablet_manager.cc:1428] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
W20260927 18:49:26.633651 12876 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:49:26.502898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:48995)
0927 18:49:26.503361 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.503416 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:26.504237 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:26.632867 (+128630us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
W20260927 18:49:26.634289 12874 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:41559 (0 consecutive failures): Network error: Failed to ping master at 127.29.159.126:41559: Client connection negotiation failed: client connection to 127.29.159.126:41559: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:49:26.635023 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.634764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:56275)
0927 18:49:26.634891 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.634962 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 18:49:26.635812 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.635598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:47589)
0927 18:49:26.635717 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.635767 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 18:49:26.636011 12874 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:49:26.664148 12877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:26.664420 12877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:26.664499 12877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:26.668682 12877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:26.668756 12877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:26.668817 12877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:49:26.674438 12877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:33857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=39223
--tserver_master_addrs=127.29.159.126:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:26.675622 12877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:26.676664 12877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:26.686496 12885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:26.686491 12884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:26.686636 12877 server_base.cc:1034] running on GCE node
W20260927 18:49:26.686866 12887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:26.687282 12877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:26.687996 12877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:26.689165 12877 hybrid_clock.cc:648] HybridClock initialized: now 1790534966689130 us; error 46 us; skew 500 ppm
Sep 27 18:49:26 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534966, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:26.691988 12877 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:49:26.692878 12877 webserver.cc:460] Webserver started at http://127.29.159.66:39223/ using document root <none> and password file <none>
I20260927 18:49:26.693411 12877 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:26.693471 12877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:26.697394 12877 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:49:26.699646 12894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:26.701031 12877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:49:26.701174 12877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "88d6b57e91a549d196c385819186d09b"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "42b9be8a1814e03539e585cc728a2e76"
server_key_iv: "b8c434d16e111f2b82844b5c216b7717"
server_key_version: "encryptionkey@0"
I20260927 18:49:26.701781 12877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:26.726218 12877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:26.729782 12877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:26.729990 12877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:26.730611 12877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:26.731532 12877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:26.731582 12877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:26.731645 12877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:26.731684 12877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:26.742604 12877 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:33857
I20260927 18:49:26.742621 13007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:33857 every 8 connection(s)
I20260927 18:49:26.743703 12877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:26.746915 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 12877
I20260927 18:49:26.747299 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:35751
--local_ip_for_outbound_sockets=127.29.159.67
--tserver_master_addrs=127.29.159.126:41559
--webserver_port=34517
--webserver_interface=127.29.159.67
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:49:26.747670 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.745895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:36809)
0927 18:49:26.746348 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.746762 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
W20260927 18:49:26.748137 13008 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:41559 (0 consecutive failures): Network error: Failed to ping master at 127.29.159.126:41559: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
I20260927 18:49:26.748819 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.748535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:52933)
0927 18:49:26.748677 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.748745 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 18:49:26.749599 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.749354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:56253)
0927 18:49:26.749463 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.749521 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 18:49:26.749781 13008 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:49:26.858585 13011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:26.858829 13011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:26.858872 13011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:26.863027 13011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:26.863085 13011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:26.863142 13011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:49:26.868139 13011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:35751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=34517
--tserver_master_addrs=127.29.159.126:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:26.869055 13011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:26.870012 13011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:26.880066 13016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:26.880062 13017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:26.880185 13019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:26.880409 13011 server_base.cc:1034] running on GCE node
I20260927 18:49:26.880775 13011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:26.881426 13011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:26.882616 13011 hybrid_clock.cc:648] HybridClock initialized: now 1790534966882575 us; error 54 us; skew 500 ppm
Sep 27 18:49:26 dist-test-slave-8tg2 krb5kdc[11563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534966, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:26.885661 13011 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:49:26.886472 13011 webserver.cc:460] Webserver started at http://127.29.159.67:34517/ using document root <none> and password file <none>
I20260927 18:49:26.887017 13011 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:26.887079 13011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:26.891247 13011 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:26.893369 13026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:26.894600 13011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:49:26.894749 13011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "88b6e3b4e1c94ffaa9b90561e127b594"
format_stamp: "Formatted at 2026-09-27 18:49:24 on dist-test-slave-8tg2"
server_key: "f8920269a0470bf5d3faa7ad8c4ff6c8"
server_key_iv: "8ef0caf9e2f071e1695f4bb6a60def6d"
server_key_version: "encryptionkey@0"
I20260927 18:49:26.895288 13011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:26.919190 13011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:26.922652 13011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:26.922873 13011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:26.923545 13011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:26.924389 13011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:26.924463 13011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:26.924528 13011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:26.924578 13011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:26.933733 13011 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:35751
I20260927 18:49:26.933795 13139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:35751 every 8 connection(s)
I20260927 18:49:26.934779 13011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:26.938124 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 13011
I20260927 18:49:26.939034 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.937128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:34893)
0927 18:49:26.937637 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.938085 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":154,"threads_started":1}
W20260927 18:49:26.939575 13140 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:41559 (0 consecutive failures): Network error: Failed to ping master at 127.29.159.126:41559: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
I20260927 18:49:26.940124 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.939906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:34043)
0927 18:49:26.940017 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.940066 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 18:49:26.940762 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:26.940560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:49545)
0927 18:49:26.940656 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:26.940711 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 18:49:26.940922 13140 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:49:27.637084 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:27.636628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:44091)
0927 18:49:27.636882 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:27.636942 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260927 18:49:27.750597 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:27.750247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:44073)
0927 18:49:27.750462 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:27.750511 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20260927 18:49:27.942013 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:27.941426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:41721)
0927 18:49:27.941675 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:27.941903 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260927 18:49:27.968731 13147 raft_consensus.cc:482] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:27.968933 13147 raft_consensus.cc:504] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:27.969928 13147 leader_election.cc:288] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857), 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751)
Sep 27 18:49:27 dist-test-slave-8tg2 krb5kdc[11563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534966, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.66@KRBTEST.COM
Sep 27 18:49:27 dist-test-slave-8tg2 krb5kdc[11563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534966, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.67@KRBTEST.COM
I20260927 18:49:27.988981 13150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:27.970698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:58409 (local address 127.29.159.67:35751)
0927 18:49:27.971031 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:27.971054 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:27.971085 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:49:27.971769 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:27.971776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:27.972165 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:27.972444 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:27.973775 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:27.974714 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:27.975959 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:27.976399 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:27.976572 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:27.984601 (+ 8029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:27.984607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:27.984619 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:27.984649 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:27.986387 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:27.986943 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:27.986950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:27.986956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:27.987085 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:27.987483 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:27.987486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:27.987488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:27.987875 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:27.988024 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:27.988542 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 18:49:27.988793 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":118,"threads_started":1}
I20260927 18:49:27.988969 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:27.970210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:33857 (local address 127.29.159.65:49539)
0927 18:49:27.970362 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:27.970379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:27.970500 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:27.972430 (+ 1930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:27.972436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:27.973042 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:27.973600 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:27.973615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:27.974853 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:27.974863 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:27.975816 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:27.975826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:27.976522 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:27.976525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:27.976732 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:27.982240 (+ 5508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:27.982253 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:27.984033 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:27.986426 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:27.986435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:27.986453 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:27.986831 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:27.987179 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:27.987185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:27.987189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:27.987336 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:27.987993 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:27.988000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:27.988235 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 18:49:27.988420 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 18:49:27.988771 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":86,"spinlock_wait_cycles":23424}
I20260927 18:49:27.989017 13148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:27.970352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:49539 (local address 127.29.159.66:33857)
0927 18:49:27.970762 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:27.970788 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:27.970838 (+ 50us) server_negotiation.cc:407] Connection header received
0927 18:49:27.971639 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:27.971646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:27.972078 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:27.972373 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:27.973757 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:27.974714 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:27.975941 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:27.976398 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:27.976577 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:27.984310 (+ 7733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:27.984319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:27.984340 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:27.984375 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:27.986284 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:27.986963 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:27.986968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:27.986972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:27.987079 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:27.987453 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:27.987459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:27.987460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:27.987875 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:27.988007 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:27.988539 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 18:49:27.988834 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":72,"threads_started":1}
I20260927 18:49:27.989012 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:27.970526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:35751 (local address 127.29.159.65:58409)
0927 18:49:27.970811 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:27.970825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:27.970950 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:27.972430 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:27.972436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:27.973041 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:27.973599 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:27.973615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:27.974853 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:27.974857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:27.975837 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:27.975846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:27.976522 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:27.976526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:27.976730 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:27.982210 (+ 5480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:27.982235 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:27.984390 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:27.986504 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:27.986506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:27.986508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:27.986823 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:27.987200 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:27.987203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:27.987205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:27.987342 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:27.988007 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:27.988010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:27.988234 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 18:49:27.988417 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 18:49:27.988853 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":359,"spinlock_wait_cycles":47104,"thread_start_us":86,"threads_started":1}
W20260927 18:49:27.991883 12762 leader_election.cc:341] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857): Not found: Tablet not found: 3def9ff332d04542b2f5c16fa10127a2
W20260927 18:49:27.992215 12762 leader_election.cc:341] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751): Not found: Tablet not found: 3def9ff332d04542b2f5c16fa10127a2
I20260927 18:49:27.992372 12762 leader_election.cc:302] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19bfd55a6c764979a211402015c5dacb; no voters: 88b6e3b4e1c94ffaa9b90561e127b594, 88d6b57e91a549d196c385819186d09b
I20260927 18:49:27.992657 13147 raft_consensus.cc:2726] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:49:28.638370 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:28.637862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:53789)
0927 18:49:28.638154 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:28.638243 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260927 18:49:28.751680 13152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:28.751261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:49357)
0927 18:49:28.751539 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:28.751596 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":74,"threads_started":1}
I20260927 18:49:28.943331 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:28.942875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:39539)
0927 18:49:28.943158 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:28.943228 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260927 18:49:29.637502 13154 raft_consensus.cc:482] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:29.637637 13154 raft_consensus.cc:504] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:29.638068 13154 leader_election.cc:288] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857), 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751)
W20260927 18:49:29.638972 12762 leader_election.cc:341] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751): Not found: Tablet not found: 3def9ff332d04542b2f5c16fa10127a2
W20260927 18:49:29.639113 12762 leader_election.cc:341] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857): Not found: Tablet not found: 3def9ff332d04542b2f5c16fa10127a2
I20260927 18:49:29.639173 12762 leader_election.cc:302] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19bfd55a6c764979a211402015c5dacb; no voters: 88b6e3b4e1c94ffaa9b90561e127b594, 88d6b57e91a549d196c385819186d09b
I20260927 18:49:29.639408 13154 raft_consensus.cc:2726] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:49:29.639478 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:29.639015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:55375)
0927 18:49:29.639266 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:29.639344 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":58,"threads_started":1}
I20260927 18:49:29.752817 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:29.752458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:54785)
0927 18:49:29.752681 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:29.752729 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":77,"threads_started":1}
I20260927 18:49:29.944507 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:29.944085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:35933)
0927 18:49:29.944339 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:29.944407 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
I20260927 18:49:30.640618 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:30.640183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:46057)
0927 18:49:30.640447 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:30.640511 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
I20260927 18:49:30.753832 13159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:30.753470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:40329)
0927 18:49:30.753695 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:30.753744 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260927 18:49:30.945618 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:30.945240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:49577)
0927 18:49:30.945481 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:30.945533 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260927 18:49:31.642057 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:31.641575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:36679)
0927 18:49:31.641898 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:31.641959 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":74,"threads_started":1}
I20260927 18:49:31.754908 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:31.754526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:53651)
0927 18:49:31.754771 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:31.754824 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
I20260927 18:49:31.946861 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:31.946446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:44829)
0927 18:49:31.946708 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:31.946773 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":113,"threads_started":1}
I20260927 18:49:32.132130 13164 raft_consensus.cc:482] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:32.132324 13164 raft_consensus.cc:504] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:32.132766 13164 leader_election.cc:288] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857), 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751)
W20260927 18:49:32.133649 12762 leader_election.cc:341] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751): Not found: Tablet not found: 3def9ff332d04542b2f5c16fa10127a2
W20260927 18:49:32.133824 12762 leader_election.cc:341] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857): Not found: Tablet not found: 3def9ff332d04542b2f5c16fa10127a2
I20260927 18:49:32.133904 12762 leader_election.cc:302] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19bfd55a6c764979a211402015c5dacb; no voters: 88b6e3b4e1c94ffaa9b90561e127b594, 88d6b57e91a549d196c385819186d09b
I20260927 18:49:32.134070 13164 raft_consensus.cc:2726] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:49:32.643225 13165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:32.642864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:33043)
0927 18:49:32.643083 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:32.643135 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":68,"threads_started":1}
I20260927 18:49:32.756106 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:32.755660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:50947)
0927 18:49:32.755959 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:32.756016 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":68,"threads_started":1}
I20260927 18:49:32.947999 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:32.947573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:38715)
0927 18:49:32.947837 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:32.947904 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260927 18:49:33.644450 13168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:33.643927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:37669)
0927 18:49:33.644249 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:33.644319 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":82,"threads_started":1}
I20260927 18:49:33.757208 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:33.756795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:41287)
0927 18:49:33.757068 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:33.757124 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":68,"threads_started":1}
I20260927 18:49:33.856823 13170 raft_consensus.cc:482] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:33.856974 13170 raft_consensus.cc:504] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:33.857394 13170 leader_election.cc:288] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857), 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751)
W20260927 18:49:33.858186 12762 leader_election.cc:341] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857): Not found: Tablet not found: 3def9ff332d04542b2f5c16fa10127a2
W20260927 18:49:33.858371 12762 leader_election.cc:341] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751): Not found: Tablet not found: 3def9ff332d04542b2f5c16fa10127a2
I20260927 18:49:33.858434 12762 leader_election.cc:302] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19bfd55a6c764979a211402015c5dacb; no voters: 88b6e3b4e1c94ffaa9b90561e127b594, 88d6b57e91a549d196c385819186d09b
I20260927 18:49:33.858626 13170 raft_consensus.cc:2726] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:49:33.949328 13171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:33.948794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:40883)
0927 18:49:33.949123 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:33.949183 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":97,"threads_started":1}
I20260927 18:49:34.645642 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:34.645216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:53843)
0927 18:49:34.645480 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:34.645541 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260927 18:49:34.758582 13173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:34.758093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:50255)
0927 18:49:34.758382 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:34.758439 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":116,"threads_started":1}
I20260927 18:49:34.950556 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:34.950098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:39985)
0927 18:49:34.950385 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:34.950448 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":72,"threads_started":1}
I20260927 18:49:35.647002 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:35.646582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:40203)
0927 18:49:35.646855 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:35.646909 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260927 18:49:35.759902 13176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:35.759392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:43263)
0927 18:49:35.759682 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:35.759794 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":99,"threads_started":1}
I20260927 18:49:35.951874 13177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:35.951341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:54333)
0927 18:49:35.951676 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:35.951740 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":162,"threads_started":1}
I20260927 18:49:36.648391 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:36.647962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:38915)
0927 18:49:36.648234 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:36.648295 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260927 18:49:36.671149 13179 raft_consensus.cc:482] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:36.671290 13179 raft_consensus.cc:504] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d6b57e91a549d196c385819186d09b" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 33857 } } peers { permanent_uuid: "88b6e3b4e1c94ffaa9b90561e127b594" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 35751 } } peers { permanent_uuid: "19bfd55a6c764979a211402015c5dacb" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46481 } }
I20260927 18:49:36.671701 13179 leader_election.cc:288] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857), 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751)
W20260927 18:49:36.672480 12762 leader_election.cc:341] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88d6b57e91a549d196c385819186d09b (127.29.159.66:33857): Not found: Tablet not found: 3def9ff332d04542b2f5c16fa10127a2
W20260927 18:49:36.672600 12762 leader_election.cc:341] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88b6e3b4e1c94ffaa9b90561e127b594 (127.29.159.67:35751): Not found: Tablet not found: 3def9ff332d04542b2f5c16fa10127a2
I20260927 18:49:36.672638 12762 leader_election.cc:302] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19bfd55a6c764979a211402015c5dacb; no voters: 88b6e3b4e1c94ffaa9b90561e127b594, 88d6b57e91a549d196c385819186d09b
I20260927 18:49:36.672763 13179 raft_consensus.cc:2726] T 3def9ff332d04542b2f5c16fa10127a2 P 19bfd55a6c764979a211402015c5dacb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:49:36.760980 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:36.760563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:50337)
0927 18:49:36.760813 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:36.760869 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":87,"threads_started":1}
I20260927 18:49:36.953504 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:36.952977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:54809)
0927 18:49:36.953309 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:36.953387 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":123,"threads_started":1}
I20260927 18:49:37.649533 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.649120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:37875)
0927 18:49:37.649382 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:37.649439 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
I20260927 18:49:37.762202 13183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.761775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:51625)
0927 18:49:37.762032 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:37.762099 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20260927 18:49:37.954600 13184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:37.954190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:52795)
0927 18:49:37.954453 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:37.954507 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260927 18:49:38.650735 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:38.650335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:35697)
0927 18:49:38.650580 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:38.650645 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260927 18:49:38.763391 13186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:38.763030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:36393)
0927 18:49:38.763244 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:38.763302 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260927 18:49:38.955807 13187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:38.955350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:38057)
0927 18:49:38.955658 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:38.955716 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":147,"threads_started":1}
I20260927 18:49:39.651961 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:39.651589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:33053)
0927 18:49:39.651818 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:39.651870 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260927 18:49:39.764551 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:39.764196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:45811)
0927 18:49:39.764417 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:39.764468 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260927 18:49:39.956940 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:39.956544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:47975)
0927 18:49:39.956779 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:39.956838 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260927 18:49:40.653052 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.652628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:38177)
0927 18:49:40.652902 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:40.652958 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":72,"threads_started":1}
I20260927 18:49:40.765739 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.765352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:44679)
0927 18:49:40.765595 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:40.765648 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260927 18:49:40.958256 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:40.957825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.67:54717)
0927 18:49:40.958097 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:40.958149 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
I20260927 18:49:41.654346 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:41.653963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.65:51553)
0927 18:49:41.654195 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:41.654248 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
I20260927 18:49:41.766884 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:41.766524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:41559 (local address 127.29.159.66:38351)
0927 18:49:41.766746 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:41.766798 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:41559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260927 18:49:41.946604 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 12710
I20260927 18:49:41.952549 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 12877
I20260927 18:49:41.957835 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 13011
2026-09-27T18:49:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19159 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:49:42 dist-test-slave-8tg2 krb5kdc[13198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:49:42 dist-test-slave-8tg2 krb5kdc[13198](info): set up 2 sockets
Sep 27 18:49:42 dist-test-slave-8tg2 krb5kdc[13198](info): commencing operation
krb5kdc: starting...
W20260927 18:49:43.007795 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:49:43 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:49:43.130259 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:36825
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:43.240048 13211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:43.240343 13211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:43.240427 13211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:43.244724 13211 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:49:43.244791 13211 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:49:43.244807 13211 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:49:43.244822 13211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:49:43.244840 13211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:43.244908 13211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:43.244936 13211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:49:43.244969 13211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:49:43.251169 13211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:43.252336 13211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:43.253458 13211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:43.262668 13216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:43.262899 13217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:43.262914 13211 server_base.cc:1034] running on GCE node
W20260927 18:49:43.262668 13219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:43.263652 13211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:43.264840 13211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:43.266016 13211 hybrid_clock.cc:648] HybridClock initialized: now 1790534983265984 us; error 46 us; skew 500 ppm
Sep 27 18:49:43 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534983, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:43.269006 13211 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:49:43.270005 13211 webserver.cc:460] Webserver started at http://127.29.159.126:43859/ using document root <none> and password file <none>
I20260927 18:49:43.270587 13211 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:43.270650 13211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:43.270902 13211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:43.273096 13211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "19d12cf4bd2f44b9af310e4fc6081192"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "916ce4dab80e6d5b5048ab8833ecc09e"
server_key_iv: "bf5071fab93259963770c0540e133625"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.273684 13211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "19d12cf4bd2f44b9af310e4fc6081192"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "916ce4dab80e6d5b5048ab8833ecc09e"
server_key_iv: "bf5071fab93259963770c0540e133625"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.277611 13211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:49:43.280026 13226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:43.281327 13211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 18:49:43.281487 13211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "19d12cf4bd2f44b9af310e4fc6081192"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "916ce4dab80e6d5b5048ab8833ecc09e"
server_key_iv: "bf5071fab93259963770c0540e133625"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.281642 13211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:43.298398 13211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:43.302003 13211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:43.302238 13211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:43.309578 13211 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:36825
I20260927 18:49:43.309868 13278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:36825 every 8 connection(s)
I20260927 18:49:43.310562 13211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:49:43.313421 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:43.316335 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 13211
I20260927 18:49:43.316464 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:49:43.316821 30333 external_mini_cluster.cc:1428] Setting key bb46cef0922447717a6281a219c6eab4
I20260927 18:49:43.320072 13279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Bootstrap starting.
I20260927 18:49:43.322789 13279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:43.323669 13279 log.cc:824] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:43.326052 13279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: No bootstrap required, opened a new log
Sep 27 18:49:43 dist-test-slave-8tg2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:43.330875 13279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:49:43.331156 13279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:43.331234 13279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d12cf4bd2f44b9af310e4fc6081192, State: Initialized, Role: FOLLOWER
I20260927 18:49:43.331738 13279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:49:43.331897 13279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:49:43.331964 13279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:49:43.331969 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:43.318255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.29.159.126:36825)
0927 18:49:43.318668 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:43.318683 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:43.318719 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:49:43.319369 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:43.319387 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:43.319771 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:43.320050 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:43.320977 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.321949 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.323123 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.323696 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.323887 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:43.326641 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:43.326647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:43.326662 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:43.326691 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:43.328807 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:43.329243 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:43.329250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:43.329256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:43.329397 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:43.329668 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:43.329672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:43.329674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:43.330212 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:43.330415 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:43.330649 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 18:49:43.330955 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":130,"threads_started":1}
I20260927 18:49:43.332080 13279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:43.333209 13279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:49:43.333572 13279 leader_election.cc:302] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d12cf4bd2f44b9af310e4fc6081192; no voters:
I20260927 18:49:43.333874 13279 leader_election.cc:288] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:49:43.333983 13284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:43.334275 13284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 1 LEADER]: Becoming Leader. State: Replica: 19d12cf4bd2f44b9af310e4fc6081192, State: Running, Role: LEADER
I20260927 18:49:43.334611 13284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:49:43.335132 13279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:49:43.336344 13286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d12cf4bd2f44b9af310e4fc6081192. Latest consensus state: current_term: 1 leader_uuid: "19d12cf4bd2f44b9af310e4fc6081192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } } }
I20260927 18:49:43.336349 13285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19d12cf4bd2f44b9af310e4fc6081192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } } }
I20260927 18:49:43.336534 13285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:43.336843 13286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: This master's current role is: LEADER
I20260927 18:49:43.336953 13293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:49:43.340153 13293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:49:43.345952 13293 catalog_manager.cc:1347] Generated new cluster ID: bbaecae441114c799b264ce688cdcb7f
I20260927 18:49:43.346029 13293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:49:43.365520 13293 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:49:43.366489 13293 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:49:43.389225 13293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Generated new TSK 0
I20260927 18:49:43.389987 13293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:49:43.427749 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:36825
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:43.546348 13305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:43.546597 13305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:43.546665 13305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:43.550673 13305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:43.550737 13305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:43.550849 13305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:49:43.556254 13305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:43.557322 13305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:43.558354 13305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:43.568034 13311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:43.568235 13305 server_base.cc:1034] running on GCE node
W20260927 18:49:43.568017 13310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:43.568017 13313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:43.568694 13305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:43.569340 13305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:43.570539 13305 hybrid_clock.cc:648] HybridClock initialized: now 1790534983570521 us; error 28 us; skew 500 ppm
Sep 27 18:49:43 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534983, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:43.573453 13305 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:49:43.574324 13305 webserver.cc:460] Webserver started at http://127.29.159.65:36135/ using document root <none> and password file <none>
I20260927 18:49:43.574905 13305 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:43.574994 13305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:43.575228 13305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:43.577353 13305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "9a2d7e4a753e4994916fbeac465284b1"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "ae59320a9425e71f0fd719e788760dfd"
server_key_iv: "0b691b59725f5173ddf541f4f3a77fbf"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.577980 13305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "9a2d7e4a753e4994916fbeac465284b1"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "ae59320a9425e71f0fd719e788760dfd"
server_key_iv: "0b691b59725f5173ddf541f4f3a77fbf"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.582072 13305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:49:43.584606 13320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:43.586061 13305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:49:43.586190 13305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "9a2d7e4a753e4994916fbeac465284b1"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "ae59320a9425e71f0fd719e788760dfd"
server_key_iv: "0b691b59725f5173ddf541f4f3a77fbf"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.586308 13305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:43.619350 13305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:43.622377 13305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:43.622586 13305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:43.623201 13305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:43.624112 13305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:43.624193 13305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:43.624269 13305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:43.624323 13305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:43.633241 13305 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:39989
I20260927 18:49:43.633275 13433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:39989 every 8 connection(s)
I20260927 18:49:43.634125 13305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:49:43.635226 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 13305
I20260927 18:49:43.635411 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:49:43.635835 30333 external_mini_cluster.cc:1428] Setting key 84731820be0fcd3525fd33cda25c27d7
Sep 27 18:49:43 dist-test-slave-8tg2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534983, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
I20260927 18:49:43.649515 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:43.636272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:57599 (local address 127.29.159.126:36825)
0927 18:49:43.636439 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:43.636443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:43.637458 (+ 1015us) server_negotiation.cc:407] Connection header received
0927 18:49:43.638779 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:43.638782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:43.638833 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:43.638913 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:43.640187 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.641087 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.642453 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.642921 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.643058 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:43.645656 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:43.645664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:43.645667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:43.645691 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:43.647219 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:43.647860 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:43.647864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:43.647866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:43.647953 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:43.648344 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:43.648347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:43.648349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:43.648558 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:43.648665 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:43.649198 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 18:49:43.649337 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:49:43.650341 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:43.636782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:57599)
0927 18:49:43.637308 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:43.637362 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:43.638563 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:43.639088 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:43.639096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:43.639540 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:43.640038 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:43.640052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.641264 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.641267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:43.642315 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:43.642325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.643090 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.643093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:43.643283 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:43.643987 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:43.644004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:43.645470 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:43.647368 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:43.647374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:43.647390 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:43.647746 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:43.648067 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:43.648070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:43.648072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:43.648226 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:43.648675 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:43.648684 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:43.648950 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 18:49:43.649143 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 18:49:43.649485 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":143,"threads_started":1}
I20260927 18:49:43.651804 13434 heartbeater.cc:347] Connected to a master server at 127.29.159.126:36825
I20260927 18:49:43.652068 13434 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:43.652587 13434 heartbeater.cc:511] Master 127.29.159.126:36825 requested a full tablet report, sending...
I20260927 18:49:43.654207 13243 ts_manager.cc:194] Registered new tserver with Master: 9a2d7e4a753e4994916fbeac465284b1 (127.29.159.65:39989)
I20260927 18:49:43.655421 13243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:57599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:49:43.667217 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:36825
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:43.779093 13439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:43.779376 13439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:43.779449 13439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:43.783710 13439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:43.783773 13439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:43.783862 13439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:49:43.789434 13439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:43.790601 13439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:43.791714 13439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:43.801782 13444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:43.801806 13445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:43.801864 13439 server_base.cc:1034] running on GCE node
W20260927 18:49:43.801769 13447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:43.802475 13439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:43.803179 13439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:43.804345 13439 hybrid_clock.cc:648] HybridClock initialized: now 1790534983804308 us; error 53 us; skew 500 ppm
Sep 27 18:49:43 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534983, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:43.807272 13439 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:49:43.808279 13439 webserver.cc:460] Webserver started at http://127.29.159.66:34745/ using document root <none> and password file <none>
I20260927 18:49:43.808835 13439 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:43.808900 13439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:43.809072 13439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:43.811102 13439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "774c258df5cc4adcb54b6f90ea831ad2"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "55893a9494a760e6f5e0463ce3c884dd"
server_key_iv: "b8734fc0fac89426312c4ef25dc2145a"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.811691 13439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "774c258df5cc4adcb54b6f90ea831ad2"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "55893a9494a760e6f5e0463ce3c884dd"
server_key_iv: "b8734fc0fac89426312c4ef25dc2145a"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.815380 13439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:49:43.818277 13454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:43.819577 13439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:49:43.819701 13439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "774c258df5cc4adcb54b6f90ea831ad2"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "55893a9494a760e6f5e0463ce3c884dd"
server_key_iv: "b8734fc0fac89426312c4ef25dc2145a"
server_key_version: "encryptionkey@0"
I20260927 18:49:43.819839 13439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:43.838162 13439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:43.841529 13439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:43.841727 13439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:43.842307 13439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:43.843115 13439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:43.843184 13439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:43.843254 13439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:43.843310 13439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:43.851929 13439 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:43761
I20260927 18:49:43.851953 13567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:43761 every 8 connection(s)
I20260927 18:49:43.852810 13439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:49:43.853658 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 13439
I20260927 18:49:43.853770 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:49:43.854076 30333 external_mini_cluster.cc:1428] Setting key 7fa310bebe8d4accdfca6c16c9e2aef7
Sep 27 18:49:43 dist-test-slave-8tg2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534983, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:49:43.867455 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:43.854824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:51915 (local address 127.29.159.126:36825)
0927 18:49:43.854949 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:43.854954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:43.855932 (+ 978us) server_negotiation.cc:407] Connection header received
0927 18:49:43.857286 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:43.857290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:43.857340 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:43.857418 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:43.858592 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.859244 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.860414 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.860648 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.860758 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:43.863175 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:43.863178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:43.863180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:43.863203 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:43.864827 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:43.865408 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:43.865412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:43.865414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:43.865508 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:43.865982 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:43.865985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:43.865987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:43.866239 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:43.866359 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:43.866994 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 18:49:43.867156 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:49:43.868342 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:43.855246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:51915)
0927 18:49:43.855790 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:43.855842 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:43.857066 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:43.857582 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:43.857594 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:43.857963 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:43.858445 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:43.858457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.859370 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.859376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:43.860289 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:43.860299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:43.860802 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:43.860806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:43.860973 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:43.861570 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:43.861585 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:43.863015 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:43.864987 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:43.864994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:43.865016 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:43.865298 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:43.865614 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:43.865617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:43.865619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:43.865848 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:43.866422 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:43.866427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:43.866707 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 18:49:43.866899 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 18:49:43.867162 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":174,"threads_started":1}
I20260927 18:49:43.869568 13568 heartbeater.cc:347] Connected to a master server at 127.29.159.126:36825
I20260927 18:49:43.869808 13568 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:43.870323 13568 heartbeater.cc:511] Master 127.29.159.126:36825 requested a full tablet report, sending...
I20260927 18:49:43.871364 13243 ts_manager.cc:194] Registered new tserver with Master: 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761)
I20260927 18:49:43.871971 13243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:51915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:49:43.882226 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:36825
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:49:43.996300 13573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:49:43.996595 13573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:49:43.996706 13573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:49:44.001211 13573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:49:44.001312 13573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:49:44.001400 13573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:49:44.006698 13573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:49:44.007884 13573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:49:44.008932 13573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:49:44.019013 13581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:49:44.019095 13579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:44.019111 13573 server_base.cc:1034] running on GCE node
W20260927 18:49:44.019021 13578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:49:44.019755 13573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:49:44.020432 13573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:49:44.021589 13573 hybrid_clock.cc:648] HybridClock initialized: now 1790534984021575 us; error 27 us; skew 500 ppm
Sep 27 18:49:44 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534984, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:49:44.024479 13573 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:49:44.025362 13573 webserver.cc:460] Webserver started at http://127.29.159.67:44371/ using document root <none> and password file <none>
I20260927 18:49:44.025920 13573 fs_manager.cc:359] Metadata directory not provided
I20260927 18:49:44.025981 13573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:49:44.026217 13573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:49:44.028285 13573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da"
format_stamp: "Formatted at 2026-09-27 18:49:44 on dist-test-slave-8tg2"
server_key: "a61b878dc5bc83d9635e998a5d91334f"
server_key_iv: "fc442182d4c8b71213c7d4d059cca5be"
server_key_version: "encryptionkey@0"
I20260927 18:49:44.028858 13573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da"
format_stamp: "Formatted at 2026-09-27 18:49:44 on dist-test-slave-8tg2"
server_key: "a61b878dc5bc83d9635e998a5d91334f"
server_key_iv: "fc442182d4c8b71213c7d4d059cca5be"
server_key_version: "encryptionkey@0"
I20260927 18:49:44.032475 13573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:49:44.034718 13588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:44.036044 13573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:49:44.036222 13573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da"
format_stamp: "Formatted at 2026-09-27 18:49:44 on dist-test-slave-8tg2"
server_key: "a61b878dc5bc83d9635e998a5d91334f"
server_key_iv: "fc442182d4c8b71213c7d4d059cca5be"
server_key_version: "encryptionkey@0"
I20260927 18:49:44.036330 13573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:49:44.070631 13573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:49:44.073616 13573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:49:44.073825 13573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:49:44.074447 13573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:49:44.075284 13573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:49:44.075356 13573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:44.075462 13573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:49:44.075520 13573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:49:44.084444 13573 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:41967
I20260927 18:49:44.084461 13701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:41967 every 8 connection(s)
I20260927 18:49:44.085399 13573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:49:44.089183 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 13573
I20260927 18:49:44.089308 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:49:44.089615 30333 external_mini_cluster.cc:1428] Setting key 8c31ada7ef96a9f34974b3a077bb1965
Sep 27 18:49:44 dist-test-slave-8tg2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534984, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:49:44.099830 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.087221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:57285 (local address 127.29.159.126:36825)
0927 18:49:44.087372 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:44.087377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:44.088352 (+ 975us) server_negotiation.cc:407] Connection header received
0927 18:49:44.089834 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:44.089837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:44.089884 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:44.089954 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:44.091545 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.092164 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.093273 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.093488 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.093567 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.095973 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:44.095977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:44.095979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:44.096004 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:44.097396 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:44.097896 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:44.097899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:44.097900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:44.097986 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:44.098775 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:44.098778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:44.098779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:44.098969 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:44.099048 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:44.099605 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 18:49:44.099708 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:49:44.100651 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.087776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:57285)
0927 18:49:44.088199 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:44.088241 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:44.089601 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:44.090300 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:44.090308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:44.090770 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:49:44.091368 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.091382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.092281 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.092283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.093167 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.093176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.093606 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.093609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.093742 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.094275 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:49:44.094290 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:49:44.095753 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:49:44.097498 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:44.097502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:44.097514 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:44.097753 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:44.098070 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:49:44.098072 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:49:44.098073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:49:44.098197 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:49:44.099070 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:49:44.099075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:49:44.099297 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 18:49:44.099503 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 18:49:44.099758 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260927 18:49:44.101745 13702 heartbeater.cc:347] Connected to a master server at 127.29.159.126:36825
I20260927 18:49:44.102005 13702 heartbeater.cc:464] Registering TS with master...
I20260927 18:49:44.102562 13702 heartbeater.cc:511] Master 127.29.159.126:36825 requested a full tablet report, sending...
I20260927 18:49:44.103875 13243 ts_manager.cc:194] Registered new tserver with Master: 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967)
I20260927 18:49:44.104511 13243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:57285
I20260927 18:49:44.115828 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:49:44.125049 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.117819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.29.159.126:36825)
0927 18:49:44.117955 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:44.117958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:44.118130 (+ 172us) server_negotiation.cc:407] Connection header received
0927 18:49:44.118257 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:44.118259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:44.118299 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:44.118358 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:49:44.119045 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.119631 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.120438 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.120640 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.120712 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.122019 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:49:44.122022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:49:44.122024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:49:44.122047 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:49:44.123577 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:44.124008 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:44.124014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:44.124017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:44.124101 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:49:44.124356 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:49:44.124358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:49:44.124360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:49:44.124578 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:49:44.124679 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:44.124842 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 18:49:44.124932 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:49:44.127876 13243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:49:44.130767 13243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:49:44.145435 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.139130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:41967 (local address 127.0.0.1:49136)
0927 18:49:44.140304 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:44.140316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:44.140409 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:44.140964 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:44.140970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:44.141006 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:44.141269 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.141277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.142730 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.142733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.143767 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.143777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.144872 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.144876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.144988 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.145026 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:49:44.145138 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 18:49:44.145270 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":107,"spinlock_wait_cycles":7168,"thread_start_us":104,"threads_started":1}
I20260927 18:49:44.145813 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.140175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.29.159.66:43761)
0927 18:49:44.140553 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:44.140564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:44.140652 (+ 88us) server_negotiation.cc:407] Connection header received
0927 18:49:44.140765 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:44.140770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:44.140944 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:44.141096 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:44.141446 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.142624 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.144220 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.144764 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.144890 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.145035 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:44.145582 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 18:49:44.145673 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":105,"threads_started":1}
I20260927 18:49:44.145486 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.139783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:43761 (local address 127.0.0.1:50748)
0927 18:49:44.140566 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:44.140577 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:44.140661 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:44.141075 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:44.141078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:44.141098 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:44.141291 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.141298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.142777 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.142780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.144104 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.144114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.145098 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.145103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.145230 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.145251 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:49:44.145315 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 18:49:44.145370 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":88,"thread_start_us":83,"threads_started":1}
I20260927 18:49:44.146136 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.139530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.29.159.67:41967)
0927 18:49:44.140267 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:44.140282 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:44.140385 (+ 103us) server_negotiation.cc:407] Connection header received
0927 18:49:44.140524 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:44.140529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:44.140728 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:44.140881 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:44.141397 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.142614 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.143887 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.144739 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.144848 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.144987 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:44.145184 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 18:49:44.145293 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":86,"threads_started":1}
I20260927 18:49:44.146813 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.139046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:39989 (local address 127.0.0.1:48422)
0927 18:49:44.139949 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:44.140001 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:44.140125 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:44.140964 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:44.140970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:44.141007 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:44.141272 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.141277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.144108 (+ 2831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.144111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.145214 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.145223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.146437 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.146440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.146526 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.146548 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:49:44.146625 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:49:44.146684 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":77,"spinlock_wait_cycles":6784,"thread_start_us":84,"threads_started":1}
I20260927 18:49:44.147022 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.139340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.29.159.65:39989)
0927 18:49:44.140119 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:44.140134 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:44.140158 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:49:44.140335 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:44.140345 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:44.140583 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:44.140738 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:44.142780 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.143944 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.145347 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.146139 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.146243 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.146374 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:44.146674 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 18:49:44.146766 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":470,"threads_started":1}
I20260927 18:49:44.148995 13636 tablet_service.cc:1467] Processing CreateTablet for tablet b8c34646b1764153a8b8ee09ebad5684 (DEFAULT_TABLE table=test-table [id=3e00b5242a6a42509a8d7333a9e458f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:44.149070 13502 tablet_service.cc:1467] Processing CreateTablet for tablet b8c34646b1764153a8b8ee09ebad5684 (DEFAULT_TABLE table=test-table [id=3e00b5242a6a42509a8d7333a9e458f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:44.150106 13636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c34646b1764153a8b8ee09ebad5684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:44.150112 13502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c34646b1764153a8b8ee09ebad5684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:44.149878 13368 tablet_service.cc:1467] Processing CreateTablet for tablet b8c34646b1764153a8b8ee09ebad5684 (DEFAULT_TABLE table=test-table [id=3e00b5242a6a42509a8d7333a9e458f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:49:44.150758 13368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c34646b1764153a8b8ee09ebad5684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:49:44.156684 13719 tablet_bootstrap.cc:492] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Bootstrap starting.
I20260927 18:49:44.157862 13720 tablet_bootstrap.cc:492] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: Bootstrap starting.
I20260927 18:49:44.158015 13721 tablet_bootstrap.cc:492] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Bootstrap starting.
I20260927 18:49:44.158921 13719 tablet_bootstrap.cc:654] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:44.160066 13721 tablet_bootstrap.cc:654] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:44.160092 13719 log.cc:824] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:44.160465 13720 tablet_bootstrap.cc:654] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: Neither blocks nor log segments found. Creating new log.
I20260927 18:49:44.161070 13721 log.cc:824] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:44.161234 13720 log.cc:824] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:49:44.162206 13719 tablet_bootstrap.cc:492] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: No bootstrap required, opened a new log
I20260927 18:49:44.162428 13719 ts_tablet_manager.cc:1397] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 18:49:44.162866 13721 tablet_bootstrap.cc:492] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: No bootstrap required, opened a new log
I20260927 18:49:44.163048 13721 ts_tablet_manager.cc:1397] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:49:44.164731 13720 tablet_bootstrap.cc:492] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: No bootstrap required, opened a new log
I20260927 18:49:44.164937 13720 ts_tablet_manager.cc:1397] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 18:49:44.165535 13719 raft_consensus.cc:348] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } }
I20260927 18:49:44.165925 13719 raft_consensus.cc:374] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:44.166033 13719 raft_consensus.cc:729] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e3f3fed48f94c129dcc9f9ae1adf1da, State: Initialized, Role: FOLLOWER
I20260927 18:49:44.166596 13719 consensus_queue.cc:260] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } }
I20260927 18:49:44.167273 13702 heartbeater.cc:503] Master 127.29.159.126:36825 was elected leader, sending a full tablet report...
I20260927 18:49:44.167244 13721 raft_consensus.cc:348] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } }
I20260927 18:49:44.167531 13721 raft_consensus.cc:374] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:44.167580 13721 raft_consensus.cc:729] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2d7e4a753e4994916fbeac465284b1, State: Initialized, Role: FOLLOWER
I20260927 18:49:44.167666 13719 ts_tablet_manager.cc:1428] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 18:49:44.167972 13721 consensus_queue.cc:260] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } }
I20260927 18:49:44.168661 13720 raft_consensus.cc:348] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } }
I20260927 18:49:44.168900 13721 ts_tablet_manager.cc:1428] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:49:44.168946 13720 raft_consensus.cc:374] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:49:44.169032 13720 raft_consensus.cc:729] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774c258df5cc4adcb54b6f90ea831ad2, State: Initialized, Role: FOLLOWER
I20260927 18:49:44.169210 13434 heartbeater.cc:503] Master 127.29.159.126:36825 was elected leader, sending a full tablet report...
I20260927 18:49:44.169430 13720 consensus_queue.cc:260] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } }
I20260927 18:49:44.170068 13568 heartbeater.cc:503] Master 127.29.159.126:36825 was elected leader, sending a full tablet report...
I20260927 18:49:44.170324 13720 ts_tablet_manager.cc:1428] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 18:49:44.336447 13703 tablet.cc:2369] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:49:44.353982 13569 tablet.cc:2369] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:49:44.385474 13435 tablet.cc:2369] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:49:44.532625 13727 raft_consensus.cc:482] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:49:44.532830 13727 raft_consensus.cc:504] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } }
I20260927 18:49:44.534020 13727 leader_election.cc:288] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a2d7e4a753e4994916fbeac465284b1 (127.29.159.65:39989), 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967)
I20260927 18:49:44.538344 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.534447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:48037 (local address 127.29.159.65:39989)
0927 18:49:44.534575 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:44.534580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:44.534781 (+ 201us) server_negotiation.cc:407] Connection header received
0927 18:49:44.534951 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:44.534954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:44.534997 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:44.535083 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:44.535375 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.536125 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.537310 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.537744 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.537876 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.537932 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:44.538102 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 18:49:44.538179 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:49:44.538347 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.534591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:59207 (local address 127.29.159.67:41967)
0927 18:49:44.534706 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:44.534710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:44.534728 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:49:44.534850 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:44.534855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:44.534904 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:44.534991 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:49:44.535270 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.536048 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.537383 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.537755 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.537885 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.537934 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:44.538155 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 18:49:44.538239 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:49:44.538358 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.534269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:41967 (local address 127.29.159.66:59207)
0927 18:49:44.534584 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:44.534600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:44.534710 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:44.535004 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:44.535007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:44.535023 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:44.535145 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.535151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.536155 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.536161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.537232 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.537259 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.537892 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.537896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.538014 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.538032 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:49:44.538127 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:49:44.538190 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":42,"thread_start_us":89,"threads_started":1}
I20260927 18:49:44.538367 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.534345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:39989 (local address 127.29.159.66:48037)
0927 18:49:44.534678 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:44.534692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:49:44.534822 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:49:44.535090 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:49:44.535093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:49:44.535109 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:49:44.535259 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.535265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.536231 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.536234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.537189 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:49:44.537198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.537868 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.537871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:49:44.537975 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.538001 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:49:44.538062 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 18:49:44.538148 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":16,"thread_start_us":87,"threads_started":1}
I20260927 18:49:44.538967 13656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c34646b1764153a8b8ee09ebad5684" candidate_uuid: "774c258df5cc4adcb54b6f90ea831ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" is_pre_election: true
I20260927 18:49:44.539028 13388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c34646b1764153a8b8ee09ebad5684" candidate_uuid: "774c258df5cc4adcb54b6f90ea831ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2d7e4a753e4994916fbeac465284b1" is_pre_election: true
I20260927 18:49:44.539317 13388 raft_consensus.cc:2454] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774c258df5cc4adcb54b6f90ea831ad2 in term 0.
I20260927 18:49:44.539316 13656 raft_consensus.cc:2454] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774c258df5cc4adcb54b6f90ea831ad2 in term 0.
I20260927 18:49:44.539858 13458 leader_election.cc:302] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 774c258df5cc4adcb54b6f90ea831ad2, 9a2d7e4a753e4994916fbeac465284b1; no voters:
I20260927 18:49:44.540136 13727 raft_consensus.cc:2781] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:49:44.540220 13727 raft_consensus.cc:482] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:49:44.540279 13727 raft_consensus.cc:3037] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:44.541431 13727 raft_consensus.cc:504] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } }
I20260927 18:49:44.541800 13727 leader_election.cc:288] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [CANDIDATE]: Term 1 election: Requested vote from peers 9a2d7e4a753e4994916fbeac465284b1 (127.29.159.65:39989), 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967)
I20260927 18:49:44.542136 13656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c34646b1764153a8b8ee09ebad5684" candidate_uuid: "774c258df5cc4adcb54b6f90ea831ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da"
I20260927 18:49:44.542150 13388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c34646b1764153a8b8ee09ebad5684" candidate_uuid: "774c258df5cc4adcb54b6f90ea831ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2d7e4a753e4994916fbeac465284b1"
I20260927 18:49:44.542230 13656 raft_consensus.cc:3037] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:44.542289 13388 raft_consensus.cc:3037] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:49:44.543202 13656 raft_consensus.cc:2454] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774c258df5cc4adcb54b6f90ea831ad2 in term 1.
I20260927 18:49:44.543354 13388 raft_consensus.cc:2454] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774c258df5cc4adcb54b6f90ea831ad2 in term 1.
I20260927 18:49:44.543515 13455 leader_election.cc:302] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3f3fed48f94c129dcc9f9ae1adf1da, 774c258df5cc4adcb54b6f90ea831ad2; no voters:
I20260927 18:49:44.543691 13727 raft_consensus.cc:2781] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:49:44.543915 13727 raft_consensus.cc:686] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 1 LEADER]: Becoming Leader. State: Replica: 774c258df5cc4adcb54b6f90ea831ad2, State: Running, Role: LEADER
I20260927 18:49:44.544226 13727 consensus_queue.cc:237] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } }
I20260927 18:49:44.547217 13243 catalog_manager.cc:5495] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66). New cstate: current_term: 1 leader_uuid: "774c258df5cc4adcb54b6f90ea831ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } health_report { overall_health: HEALTHY } } }
I20260927 18:49:44.602056 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.597365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.29.159.66:43761)
0927 18:49:44.597534 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:49:44.597539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:49:44.597557 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:49:44.597661 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:49:44.597664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:49:44.597724 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:49:44.597854 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:49:44.598116 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.599325 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.600464 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:49:44.600715 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:49:44.600841 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:49:44.601183 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:49:44.601618 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:49:44.601767 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:49:44.601841 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 18:49:44.601900 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:49:44.607806 13388 raft_consensus.cc:1261] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Refusing update from remote peer 774c258df5cc4adcb54b6f90ea831ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:44.607806 13656 raft_consensus.cc:1261] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 1 FOLLOWER]: Refusing update from remote peer 774c258df5cc4adcb54b6f90ea831ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:49:44.608490 13727 consensus_queue.cc:1035] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:44.608659 13730 consensus_queue.cc:1035] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:49:44.616369 13735 mvcc.cc:205] Tried to move back new op lower bound from 7334031296946307072 to 7334031296695619584. Current Snapshot: MvccSnapshot[applied={T|T < 7334031296946307072}]
I20260927 18:49:44.623355 13242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:44.623555 13242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:49:44.626056 13242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Sending DeleteTablet for 3 replicas of tablet b8c34646b1764153a8b8ee09ebad5684
I20260927 18:49:44.626875 13368 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c34646b1764153a8b8ee09ebad5684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:44 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:48422
I20260927 18:49:44.627065 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 13305
I20260927 18:49:44.627173 13502 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c34646b1764153a8b8ee09ebad5684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:44 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:50748
I20260927 18:49:44.627774 13636 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c34646b1764153a8b8ee09ebad5684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:44 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:49136
I20260927 18:49:44.628705 13744 tablet_replica.cc:330] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: stopping tablet replica
I20260927 18:49:44.629042 13744 raft_consensus.cc:2229] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:49:44.629056 13743 tablet_replica.cc:330] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: stopping tablet replica
I20260927 18:49:44.629333 13743 raft_consensus.cc:2229] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:49:44.629482 13744 raft_consensus.cc:2258] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:44.629664 13743 raft_consensus.cc:2258] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:49:44.631103 13743 ts_tablet_manager.cc:1905] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:44.632567 13744 ts_tablet_manager.cc:1905] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:49:44.634501 13743 ts_tablet_manager.cc:1918] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:44.634598 13743 log.cc:1196] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/b8c34646b1764153a8b8ee09ebad5684
W20260927 18:49:44.634783 13230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:39989 (error 108)
I20260927 18:49:44.634933 13743 ts_tablet_manager.cc:1939] T b8c34646b1764153a8b8ee09ebad5684 P 774c258df5cc4adcb54b6f90ea831ad2: Deleting consensus metadata
I20260927 18:49:44.635772 13744 ts_tablet_manager.cc:1918] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:49:44.635936 13744 log.cc:1196] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/b8c34646b1764153a8b8ee09ebad5684
I20260927 18:49:44.636030 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 13439
I20260927 18:49:44.636137 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:49:44.635814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:39989 (local address 127.0.0.1:48430)
0927 18:49:44.635936 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 18:49:44.636034 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 18:49:44.636176 13227 catalog_manager.cc:4841] TS 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761): tablet b8c34646b1764153a8b8ee09ebad5684 (table test-table [id=3e00b5242a6a42509a8d7333a9e458f7]) successfully deleted
I20260927 18:49:44.636453 13744 ts_tablet_manager.cc:1939] T b8c34646b1764153a8b8ee09ebad5684 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Deleting consensus metadata
I20260927 18:49:44.637672 13227 catalog_manager.cc:4841] TS 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967): tablet b8c34646b1764153a8b8ee09ebad5684 (table test-table [id=3e00b5242a6a42509a8d7333a9e458f7]) successfully deleted
W20260927 18:49:44.639729 13230 catalog_manager.cc:4584] TS 9a2d7e4a753e4994916fbeac465284b1 (127.29.159.65:39989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8c34646b1764153a8b8ee09ebad5684: Network error: Client connection negotiation failed: client connection to 127.29.159.65:39989: connect: Connection refused (error 111)
I20260927 18:49:44.642701 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 13573
I20260927 18:49:44.647913 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 13211
I20260927 18:50:04.654312 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:36825
--webserver_interface=127.29.159.126
--webserver_port=43859
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:04.767355 13746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:04.767693 13746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:04.767735 13746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:04.771799 13746 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:50:04.771852 13746 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:50:04.771867 13746 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:50:04.771905 13746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:04.771943 13746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:04.771974 13746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:04.772014 13746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:04.772053 13746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:04.777361 13746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=43859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:04.778475 13746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:04.779533 13746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:50:04.792655 13746 server_base.cc:1034] running on GCE node
W20260927 18:50:04.792682 13752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:04.792611 13753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:04.792886 13755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:04.793499 13746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:04.794646 13746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:04.795831 13746 hybrid_clock.cc:648] HybridClock initialized: now 1790535004795807 us; error 38 us; skew 500 ppm
Sep 27 18:50:04 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535004, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:04.798799 13746 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:50:04.799763 13746 webserver.cc:460] Webserver started at http://127.29.159.126:43859/ using document root <none> and password file <none>
I20260927 18:50:04.800367 13746 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:04.800480 13746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:04.804385 13746 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:04.806592 13762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:04.807977 13746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:50:04.808118 13746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "19d12cf4bd2f44b9af310e4fc6081192"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "916ce4dab80e6d5b5048ab8833ecc09e"
server_key_iv: "bf5071fab93259963770c0540e133625"
server_key_version: "encryptionkey@0"
I20260927 18:50:04.808732 13746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:04.826004 13746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:04.828912 13746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:04.829129 13746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:04.836470 13746 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:36825
I20260927 18:50:04.836767 13814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:36825 every 8 connection(s)
I20260927 18:50:04.837663 13746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:04.840765 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 13746
I20260927 18:50:04.841221 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:39989
--local_ip_for_outbound_sockets=127.29.159.65
--tserver_master_addrs=127.29.159.126:36825
--webserver_port=36135
--webserver_interface=127.29.159.65
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:50:04.842157 13815 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:50:04.843891 13815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Bootstrap starting.
I20260927 18:50:04.860785 13815 log.cc:824] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:04.869431 13815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:50:04.869829 13815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Bootstrap complete.
I20260927 18:50:04.873948 13815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:50:04.874377 13815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d12cf4bd2f44b9af310e4fc6081192, State: Initialized, Role: FOLLOWER
I20260927 18:50:04.874866 13815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:50:04.875016 13815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:04.875098 13815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:04.875234 13815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:50:04.877197 13815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:50:04.877545 13815 leader_election.cc:302] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d12cf4bd2f44b9af310e4fc6081192; no voters:
I20260927 18:50:04.877884 13815 leader_election.cc:288] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:50:04.877967 13819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:50:04.878206 13819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 2 LEADER]: Becoming Leader. State: Replica: 19d12cf4bd2f44b9af310e4fc6081192, State: Running, Role: LEADER
I20260927 18:50:04.878578 13819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:50:04.879155 13815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:04.880108 13821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d12cf4bd2f44b9af310e4fc6081192. Latest consensus state: current_term: 2 leader_uuid: "19d12cf4bd2f44b9af310e4fc6081192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } } }
I20260927 18:50:04.880229 13821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:04.880587 13820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "19d12cf4bd2f44b9af310e4fc6081192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } } }
I20260927 18:50:04.880623 13828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:04.880681 13820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:04.885627 13828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:04.886650 13828 catalog_manager.cc:1259] Loaded cluster ID: bbaecae441114c799b264ce688cdcb7f
I20260927 18:50:04.886790 13828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:04.889453 13828 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:04.890228 13828 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Loaded TSK: 0
I20260927 18:50:04.907485 13828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Generated new TSK 1
I20260927 18:50:04.907934 13828 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:50:04.960625 13817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:04.960947 13817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:04.961023 13817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:04.965731 13817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:04.965860 13817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:04.965962 13817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:50:04.971581 13817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:39989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=36135
--tserver_master_addrs=127.29.159.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:04.972730 13817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:04.973800 13817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:04.984838 13842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:04.984851 13843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:04.985014 13845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:04.985411 13817 server_base.cc:1034] running on GCE node
I20260927 18:50:04.985865 13817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:04.986653 13817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:04.987862 13817 hybrid_clock.cc:648] HybridClock initialized: now 1790535004987824 us; error 46 us; skew 500 ppm
Sep 27 18:50:04 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535004, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:04.991602 13817 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:50:04.992735 13817 webserver.cc:460] Webserver started at http://127.29.159.65:36135/ using document root <none> and password file <none>
I20260927 18:50:04.993455 13817 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:04.993557 13817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:04.999081 13817 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:50:05.001789 13852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:05.003321 13817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 18:50:05.003482 13817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "9a2d7e4a753e4994916fbeac465284b1"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "ae59320a9425e71f0fd719e788760dfd"
server_key_iv: "0b691b59725f5173ddf541f4f3a77fbf"
server_key_version: "encryptionkey@0"
I20260927 18:50:05.004210 13817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:05.025954 13817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:05.029601 13817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:05.029829 13817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:05.030521 13817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:05.031822 13859 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:50:05.035926 13817 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:50:05.035996 13817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 18:50:05.036057 13817 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:50:05.038286 13817 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:50:05.038357 13817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 18:50:05.038666 13859 tablet_bootstrap.cc:492] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Bootstrap starting.
I20260927 18:50:05.048870 13817 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:39989
I20260927 18:50:05.049089 13966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:39989 every 8 connection(s)
I20260927 18:50:05.050205 13817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:50:05.058024 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 13817
I20260927 18:50:05.058559 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:43761
--local_ip_for_outbound_sockets=127.29.159.66
--tserver_master_addrs=127.29.159.126:36825
--webserver_port=34745
--webserver_interface=127.29.159.66
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:50:05.058849 13859 log.cc:824] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:50:05 dist-test-slave-8tg2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535004, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:05.067999 13859 tablet_bootstrap.cc:492] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:50:05.068531 13859 tablet_bootstrap.cc:492] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Bootstrap complete.
I20260927 18:50:05.070106 13859 ts_tablet_manager.cc:1397] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Time spent bootstrapping tablet: real 0.032s user 0.030s sys 0.000s
I20260927 18:50:05.070853 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.052575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:42729)
0927 18:50:05.053273 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:05.053334 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:05.054448 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:05.055776 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:05.055784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:05.056382 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:05.057128 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.057150 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.058583 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.058586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.059938 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.059952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.061003 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.061007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.061188 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.061819 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:05.061835 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:05.063893 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:05.067591 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:05.067597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:05.067610 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:05.067891 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:05.068337 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:05.068340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:05.068342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:05.068484 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:05.069086 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:05.069092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:05.069346 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 18:50:05.069554 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 18:50:05.069920 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":165,"threads_started":1}
I20260927 18:50:05.070851 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.052658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:42729 (local address 127.29.159.126:36825)
0927 18:50:05.053708 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.053728 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.053773 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:50:05.054915 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.054929 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.055308 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.055688 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:05.057317 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.058410 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.060137 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.060828 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.061015 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.064222 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:05.064232 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:05.064249 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:05.064318 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:05.067460 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.068016 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.068022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.068027 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.068209 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.068589 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.068592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.068593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.068971 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:05.069112 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.069681 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.069939 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":119,"threads_started":1}
I20260927 18:50:05.073619 13967 heartbeater.cc:347] Connected to a master server at 127.29.159.126:36825
I20260927 18:50:05.073863 13967 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:05.073877 13859 raft_consensus.cc:348] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } }
I20260927 18:50:05.074353 13967 heartbeater.cc:511] Master 127.29.159.126:36825 requested a full tablet report, sending...
I20260927 18:50:05.074414 13859 raft_consensus.cc:729] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2d7e4a753e4994916fbeac465284b1, State: Initialized, Role: FOLLOWER
I20260927 18:50:05.074955 13859 consensus_queue.cc:260] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } } peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } }
I20260927 18:50:05.077114 13859 ts_tablet_manager.cc:1428] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:50:05.077556 13779 ts_manager.cc:194] Registered new tserver with Master: 9a2d7e4a753e4994916fbeac465284b1 (127.29.159.65:39989)
I20260927 18:50:05.081125 13779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:42729
I20260927 18:50:05.083814 13967 heartbeater.cc:503] Master 127.29.159.126:36825 was elected leader, sending a full tablet report...
Sep 27 18:50:05 dist-test-slave-8tg2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535004, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.65@KRBTEST.COM
I20260927 18:50:05.091935 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.081332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.29.159.65:39989)
0927 18:50:05.082053 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.082062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.082082 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:50:05.082138 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.082142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.082354 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.082490 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:05.083234 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.084272 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.085483 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.085817 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.085901 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.088303 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:05.088320 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:05.088330 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:05.088352 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:05.089884 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.090421 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.090427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.090429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.090515 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.090817 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.090820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.090822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.091066 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:05.091208 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.091600 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.091767 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":619,"thread_start_us":111,"threads_started":1}
I20260927 18:50:05.091935 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.080501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:39989 (local address 127.0.0.1:42726)
0927 18:50:05.081270 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:05.081319 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:05.081444 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:05.082521 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:05.082527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:05.082967 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:05.083099 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.083107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.084401 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.084405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.085360 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.085372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.085933 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.085937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.086044 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.086592 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:05.086607 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:05.088166 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:05.090035 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:05.090044 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:05.090050 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:05.090296 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:05.090615 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:05.090618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:05.090620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:05.090707 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:05.091193 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:05.091199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:05.091339 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 18:50:05.091518 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 18:50:05.091734 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":148,"threads_started":1}
I20260927 18:50:05.093242 13901 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c34646b1764153a8b8ee09ebad5684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:49:44 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:42726
I20260927 18:50:05.094228 13976 tablet_replica.cc:330] stopping tablet replica
I20260927 18:50:05.094520 13976 raft_consensus.cc:2229] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:05.094766 13976 raft_consensus.cc:2258] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:05.095830 13976 ts_tablet_manager.cc:1905] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:05.099532 13976 ts_tablet_manager.cc:1918] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:05.099644 13976 log.cc:1196] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal/wals/b8c34646b1764153a8b8ee09ebad5684
I20260927 18:50:05.099929 13976 ts_tablet_manager.cc:1939] T b8c34646b1764153a8b8ee09ebad5684 P 9a2d7e4a753e4994916fbeac465284b1: Deleting consensus metadata
I20260927 18:50:05.101622 13766 catalog_manager.cc:4841] TS 9a2d7e4a753e4994916fbeac465284b1 (127.29.159.65:39989): tablet b8c34646b1764153a8b8ee09ebad5684 (table test-table [id=3e00b5242a6a42509a8d7333a9e458f7]) successfully deleted
W20260927 18:50:05.180327 13972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:05.180572 13972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:05.180614 13972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:05.184620 13972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:05.184677 13972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:05.184737 13972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:50:05.190016 13972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:43761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=34745
--tserver_master_addrs=127.29.159.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:05.191002 13972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:05.192039 13972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:05.201788 13984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:05.201963 13981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:05.201917 13982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:05.202097 13972 server_base.cc:1034] running on GCE node
I20260927 18:50:05.202553 13972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:05.203207 13972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:05.204372 13972 hybrid_clock.cc:648] HybridClock initialized: now 1790535005204377 us; error 57 us; skew 500 ppm
Sep 27 18:50:05 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535005, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:05.207202 13972 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:50:05.208240 13972 webserver.cc:460] Webserver started at http://127.29.159.66:34745/ using document root <none> and password file <none>
I20260927 18:50:05.208815 13972 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:05.208889 13972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:05.212821 13972 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:50:05.215013 13991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:05.216346 13972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:50:05.216437 13972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "774c258df5cc4adcb54b6f90ea831ad2"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "55893a9494a760e6f5e0463ce3c884dd"
server_key_iv: "b8734fc0fac89426312c4ef25dc2145a"
server_key_version: "encryptionkey@0"
I20260927 18:50:05.216976 13972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:05.246731 13972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:05.250170 13972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:05.250383 13972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:05.251001 13972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:05.251857 13972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:05.251906 13972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:05.251947 13972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:05.252000 13972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:05.260891 13972 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:43761
I20260927 18:50:05.260918 14104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:43761 every 8 connection(s)
I20260927 18:50:05.261883 13972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:50:05.268201 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 13972
I20260927 18:50:05.268699 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:41967
--local_ip_for_outbound_sockets=127.29.159.67
--tserver_master_addrs=127.29.159.126:36825
--webserver_port=44371
--webserver_interface=127.29.159.67
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:50:05 dist-test-slave-8tg2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535005, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:05.277086 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.263713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:33087 (local address 127.29.159.126:36825)
0927 18:50:05.263891 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.263896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.264792 (+ 896us) server_negotiation.cc:407] Connection header received
0927 18:50:05.266025 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.266027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.266075 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.266153 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:05.267345 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.267965 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.269723 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.269962 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.270094 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.272740 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:05.272746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:05.272751 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:05.272788 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:05.274623 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.275163 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.275168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.275173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.275274 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.275696 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.275718 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.275721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.276018 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:05.276145 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.276659 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.276826 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:50:05.277797 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.264157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:33087)
0927 18:50:05.264631 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:05.264674 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:05.265818 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:05.266327 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:05.266334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:05.266714 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:05.267197 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.267211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.268100 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.268106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.269580 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.269595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.270138 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.270143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.270324 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.270974 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:05.270990 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:05.272547 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:05.274759 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:05.274764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:05.274778 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:05.275053 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:05.275377 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:05.275380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:05.275381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:05.275596 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:05.276140 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:05.276145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:05.276389 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 18:50:05.276572 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 18:50:05.276840 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
I20260927 18:50:05.279106 14105 heartbeater.cc:347] Connected to a master server at 127.29.159.126:36825
I20260927 18:50:05.279459 14105 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:05.280203 14105 heartbeater.cc:511] Master 127.29.159.126:36825 requested a full tablet report, sending...
I20260927 18:50:05.281466 13779 ts_manager.cc:194] Registered new tserver with Master: 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761)
I20260927 18:50:05.282264 13779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:33087
W20260927 18:50:05.382241 14108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:05.382475 14108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:05.382515 14108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:05.386391 14108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:05.386443 14108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:05.386498 14108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:50:05.391512 14108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:41967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=44371
--tserver_master_addrs=127.29.159.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:05.392408 14108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:05.393306 14108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:05.402666 14114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:05.402653 14113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:05.402688 14116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:05.402871 14108 server_base.cc:1034] running on GCE node
I20260927 18:50:05.403291 14108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:05.403954 14108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:05.405133 14108 hybrid_clock.cc:648] HybridClock initialized: now 1790535005405099 us; error 45 us; skew 500 ppm
Sep 27 18:50:05 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535005, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:05.408052 14108 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:50:05.409061 14108 webserver.cc:460] Webserver started at http://127.29.159.67:44371/ using document root <none> and password file <none>
I20260927 18:50:05.409672 14108 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:05.409740 14108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:05.413676 14108 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:05.415906 14123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:05.417155 14108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:05.417286 14108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da"
format_stamp: "Formatted at 2026-09-27 18:49:44 on dist-test-slave-8tg2"
server_key: "a61b878dc5bc83d9635e998a5d91334f"
server_key_iv: "fc442182d4c8b71213c7d4d059cca5be"
server_key_version: "encryptionkey@0"
I20260927 18:50:05.417889 14108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:05.450762 14108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:05.453914 14108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:05.454139 14108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:05.454741 14108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:05.455648 14108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:05.455727 14108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:05.455822 14108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:05.455873 14108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:05.465173 14108 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:41967
I20260927 18:50:05.465191 14236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:41967 every 8 connection(s)
I20260927 18:50:05.466205 14108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
Sep 27 18:50:05 dist-test-slave-8tg2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535005, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:05.475984 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 14108
I20260927 18:50:05.479887 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.467996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:36269 (local address 127.29.159.126:36825)
0927 18:50:05.468131 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.468135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.468899 (+ 764us) server_negotiation.cc:407] Connection header received
0927 18:50:05.470107 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.470110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.470158 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.470232 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:05.471404 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.471934 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.473012 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.473213 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.473283 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.475673 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:05.475676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:05.475694 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:05.475728 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:05.477579 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.478189 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.478191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.478193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.478290 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.478632 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.478635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.478637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.478856 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:05.478957 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.479572 (+ 615us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.479668 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:50:05.480926 14239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.468318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:36269)
0927 18:50:05.468737 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:05.468780 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:05.469919 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:05.470461 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:05.470468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:05.470818 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:05.471280 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.471293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.472040 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.472043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.472908 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.472923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.473367 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.473369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.473499 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.474028 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:05.474043 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:05.475528 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:05.477697 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:05.477705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:05.477720 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:05.478087 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:05.478394 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:05.478397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:05.478399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:05.478539 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:05.478986 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:05.478991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:05.479296 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 18:50:05.479535 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 18:50:05.479905 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260927 18:50:05.482311 14237 heartbeater.cc:347] Connected to a master server at 127.29.159.126:36825
I20260927 18:50:05.482565 14237 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:05.483063 14237 heartbeater.cc:511] Master 127.29.159.126:36825 requested a full tablet report, sending...
I20260927 18:50:05.484330 13779 ts_manager.cc:194] Registered new tserver with Master: 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967)
I20260927 18:50:05.485059 13779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:36269
I20260927 18:50:05.485464 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.476744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.29.159.126:36825)
0927 18:50:05.476954 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.476958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.477148 (+ 190us) server_negotiation.cc:407] Connection header received
0927 18:50:05.477226 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.477231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.477276 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.477384 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:05.478350 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.479034 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.480258 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.480594 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.480679 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.482169 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:05.482173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:05.482175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:05.482200 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:05.483809 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.484287 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.484290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.484292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.484378 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.484675 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.484678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.484680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.484861 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:05.485069 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.485180 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.485297 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260927 18:50:05.487566 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:50:05.491245 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.488156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.29.159.126:36825)
0927 18:50:05.488344 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.488347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.488391 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:50:05.488517 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.488519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.488605 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.488691 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:05.488857 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.489382 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.490169 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.490464 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.490552 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.490845 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:05.490962 (+ 117us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 18:50:05.491160 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42288: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 18:50:05.491385 14240 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42288: token expired
W20260927 18:50:05.491719 13707 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.29.159.126:36825)
I20260927 18:50:05.499070 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.492184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.29.159.126:36825)
0927 18:50:05.492294 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.492297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.492308 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:50:05.492340 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.492342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.492372 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.492439 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:05.492992 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.493478 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.494219 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.494435 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.494508 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.495902 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:05.495906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:05.495908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:05.495930 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:05.497711 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.498101 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.498103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.498105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.498182 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:05.498423 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:05.498425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:05.498427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:05.498593 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:05.498699 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.498861 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.498956 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:50:05.501639 13779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:50:05.505241 13779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:50:05.513084 13901 tablet_service.cc:1467] Processing CreateTablet for tablet e7159cac07c9445296a2a5f95f171a81 (DEFAULT_TABLE table=test-table [id=10b6347e0b7a42acabe0f0ceec0901cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:05.513773 13901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7159cac07c9445296a2a5f95f171a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:05.516908 13859 tablet_bootstrap.cc:492] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: Bootstrap starting.
I20260927 18:50:05.517874 13859 tablet_bootstrap.cc:654] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:05.518057 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.512377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:43761 (local address 127.0.0.1:39798)
0927 18:50:05.512659 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:05.512671 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:05.512772 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:05.513706 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:05.513709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:05.513728 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:05.513874 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.513881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.515037 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.515042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.516280 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.516291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.517553 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.517557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.517659 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.517711 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 18:50:05.517786 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:50:05.517881 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"spinlock_wait_cycles":8192,"thread_start_us":140,"threads_started":1}
I20260927 18:50:05.518289 14248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.512812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.29.159.66:43761)
0927 18:50:05.513155 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.513164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.513184 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:50:05.513249 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.513254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.513544 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.513719 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:05.513971 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.514915 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.516423 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.517410 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.517520 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.517667 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.518024 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.518138 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":146,"threads_started":1}
I20260927 18:50:05.518824 14247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.512410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.29.159.67:41967)
0927 18:50:05.513295 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.513312 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.513335 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:50:05.513414 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.513419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.513627 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.513775 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:05.514206 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.515299 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.516746 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.517685 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.517798 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.517931 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.518541 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.518649 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":127,"threads_started":1}
I20260927 18:50:05.519213 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.512064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:41967 (local address 127.0.0.1:50556)
0927 18:50:05.512653 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:05.512670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:05.512762 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:05.513966 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:05.513969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:05.513987 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:05.514078 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.514084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.515455 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.515459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.516613 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.516622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.517805 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.517812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.518013 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.518043 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:50:05.519016 (+ 973us) client_negotiation.cc:240] Negotiation successful
0927 18:50:05.519099 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"spinlock_wait_cycles":3712}
I20260927 18:50:05.520123 13859 tablet_bootstrap.cc:492] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: No bootstrap required, opened a new log
I20260927 18:50:05.520210 13859 ts_tablet_manager.cc:1397] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:50:05.520654 13859 raft_consensus.cc:348] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:05.520797 13859 raft_consensus.cc:374] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:05.520839 13859 raft_consensus.cc:729] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2d7e4a753e4994916fbeac465284b1, State: Initialized, Role: FOLLOWER
I20260927 18:50:05.521056 13859 consensus_queue.cc:260] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:05.522222 14039 tablet_service.cc:1467] Processing CreateTablet for tablet e7159cac07c9445296a2a5f95f171a81 (DEFAULT_TABLE table=test-table [id=10b6347e0b7a42acabe0f0ceec0901cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:05.522439 14171 tablet_service.cc:1467] Processing CreateTablet for tablet e7159cac07c9445296a2a5f95f171a81 (DEFAULT_TABLE table=test-table [id=10b6347e0b7a42acabe0f0ceec0901cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:05.523173 13859 ts_tablet_manager.cc:1428] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 18:50:05.523578 14171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7159cac07c9445296a2a5f95f171a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:05.523589 14039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7159cac07c9445296a2a5f95f171a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:05.530191 14250 tablet_bootstrap.cc:492] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Bootstrap starting.
I20260927 18:50:05.530956 14251 tablet_bootstrap.cc:492] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2: Bootstrap starting.
I20260927 18:50:05.532953 14250 tablet_bootstrap.cc:654] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:05.533205 14251 tablet_bootstrap.cc:654] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:05.533806 14250 log.cc:824] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:05.533850 14251 log.cc:824] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:05.535611 14251 tablet_bootstrap.cc:492] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2: No bootstrap required, opened a new log
I20260927 18:50:05.535640 14250 tablet_bootstrap.cc:492] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da: No bootstrap required, opened a new log
I20260927 18:50:05.535823 14251 ts_tablet_manager.cc:1397] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:50:05.535883 14250 ts_tablet_manager.cc:1397] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:50:05.538895 14251 raft_consensus.cc:348] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:05.539114 14251 raft_consensus.cc:374] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:05.539147 14251 raft_consensus.cc:729] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774c258df5cc4adcb54b6f90ea831ad2, State: Initialized, Role: FOLLOWER
I20260927 18:50:05.539132 14250 raft_consensus.cc:348] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:05.539460 14250 raft_consensus.cc:374] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:05.539530 14250 raft_consensus.cc:729] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e3f3fed48f94c129dcc9f9ae1adf1da, State: Initialized, Role: FOLLOWER
I20260927 18:50:05.539659 14251 consensus_queue.cc:260] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:05.539997 14250 consensus_queue.cc:260] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:05.540364 14105 heartbeater.cc:503] Master 127.29.159.126:36825 was elected leader, sending a full tablet report...
I20260927 18:50:05.540663 14251 ts_tablet_manager.cc:1428] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:50:05.540838 14250 ts_tablet_manager.cc:1428] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:50:05.541221 14237 heartbeater.cc:503] Master 127.29.159.126:36825 was elected leader, sending a full tablet report...
W20260927 18:50:05.551654 13968 tablet.cc:2369] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:50:05.617954 14254 raft_consensus.cc:482] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:05.618301 14255 raft_consensus.cc:482] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:05.618242 14254 raft_consensus.cc:504] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:05.618495 14255 raft_consensus.cc:504] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:05.619511 14255 leader_election.cc:288] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761), 9a2d7e4a753e4994916fbeac465284b1 (127.29.159.65:39989)
I20260927 18:50:05.619809 14254 leader_election.cc:288] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967), 9a2d7e4a753e4994916fbeac465284b1 (127.29.159.65:39989)
I20260927 18:50:05.624357 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.619708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:41967 (local address 127.29.159.66:40301)
0927 18:50:05.619958 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:05.619972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:05.620058 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:05.620508 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:05.620511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:05.620530 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:05.620683 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.620690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.621702 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.621705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.622900 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.622909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.623888 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.623892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.623994 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.624062 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 18:50:05.624138 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 18:50:05.624208 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":34}
I20260927 18:50:05.624617 14239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.619718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:43761 (local address 127.29.159.67:36801)
0927 18:50:05.619898 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:05.619915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:05.620004 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:05.620833 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:05.620836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:05.620870 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:05.621025 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.621034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.622050 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.622053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.623219 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.623229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.624236 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.624240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.624335 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.624355 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:50:05.624417 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 18:50:05.624492 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"spinlock_wait_cycles":2304}
I20260927 18:50:05.624791 14247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.620004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:40301 (local address 127.29.159.67:41967)
0927 18:50:05.620153 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.620158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.620168 (+ 10us) server_negotiation.cc:407] Connection header received
0927 18:50:05.620210 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.620213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.620313 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.620392 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:05.620803 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.621577 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.623034 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.623635 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.623780 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.623836 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.624603 (+ 767us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.624668 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 18:50:05.625042 14248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.620416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:36801 (local address 127.29.159.66:43761)
0927 18:50:05.620527 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.620531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.620544 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:50:05.620587 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.620589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.620636 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.620729 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:05.621146 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.621936 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.623416 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.624047 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.624140 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.624183 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.624855 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.624922 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":50}
I20260927 18:50:05.625265 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.620209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:39989 (local address 127.29.159.67:35487)
0927 18:50:05.620821 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:05.620833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:05.620959 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:05.621493 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:05.621495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:05.621512 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:05.621610 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.621616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.622683 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.622686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.623877 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.623887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.624854 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.624857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.624965 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.624986 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:50:05.625056 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 18:50:05.625126 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"spinlock_wait_cycles":1280,"thread_start_us":83,"threads_started":1}
I20260927 18:50:05.625314 14191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7159cac07c9445296a2a5f95f171a81" candidate_uuid: "774c258df5cc4adcb54b6f90ea831ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" is_pre_election: true
I20260927 18:50:05.625499 14059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7159cac07c9445296a2a5f95f171a81" candidate_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774c258df5cc4adcb54b6f90ea831ad2" is_pre_election: true
I20260927 18:50:05.625526 14191 raft_consensus.cc:2454] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774c258df5cc4adcb54b6f90ea831ad2 in term 0.
I20260927 18:50:05.625586 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.620503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:35487 (local address 127.29.159.65:39989)
0927 18:50:05.621219 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.621223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.621235 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:50:05.621274 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.621277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.621316 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.621397 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:05.621754 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.622567 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.624024 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.624622 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.624740 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.624795 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.625368 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.625462 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":85,"threads_started":1}
I20260927 18:50:05.625661 14059 raft_consensus.cc:2454] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3f3fed48f94c129dcc9f9ae1adf1da in term 0.
I20260927 18:50:05.625990 13992 leader_election.cc:302] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3f3fed48f94c129dcc9f9ae1adf1da, 774c258df5cc4adcb54b6f90ea831ad2; no voters:
I20260927 18:50:05.626081 13921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7159cac07c9445296a2a5f95f171a81" candidate_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2d7e4a753e4994916fbeac465284b1" is_pre_election: true
I20260927 18:50:05.626276 14124 leader_election.cc:302] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3f3fed48f94c129dcc9f9ae1adf1da, 774c258df5cc4adcb54b6f90ea831ad2; no voters:
I20260927 18:50:05.626372 14256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.619941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:50781 (local address 127.29.159.65:39989)
0927 18:50:05.620327 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.620332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.620622 (+ 290us) server_negotiation.cc:407] Connection header received
0927 18:50:05.620830 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.620833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.620904 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.621001 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:05.621501 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.622277 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.623812 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.624447 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.624556 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.624701 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.626150 (+ 1449us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.626226 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":307,"thread_start_us":75,"threads_started":1}
I20260927 18:50:05.626346 13921 raft_consensus.cc:2454] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3f3fed48f94c129dcc9f9ae1adf1da in term 0.
I20260927 18:50:05.626456 14254 raft_consensus.cc:2781] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:50:05.626531 14254 raft_consensus.cc:482] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:50:05.626544 14255 raft_consensus.cc:2781] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:50:05.626580 14254 raft_consensus.cc:3037] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:05.626606 14255 raft_consensus.cc:482] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:50:05.626657 14255 raft_consensus.cc:3037] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:05.626748 14257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.619708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:39989 (local address 127.29.159.66:50781)
0927 18:50:05.620510 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:05.620523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:05.620695 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:05.621135 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:05.621137 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:05.621153 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:05.621379 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.621385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.622465 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.622468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.623653 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:05.623664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.625875 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.625881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:05.625995 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.626024 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:50:05.626514 (+ 490us) client_negotiation.cc:240] Negotiation successful
0927 18:50:05.626587 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":537,"thread_start_us":119,"threads_started":1}
I20260927 18:50:05.627573 13921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7159cac07c9445296a2a5f95f171a81" candidate_uuid: "774c258df5cc4adcb54b6f90ea831ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2d7e4a753e4994916fbeac465284b1" is_pre_election: true
I20260927 18:50:05.627712 13921 raft_consensus.cc:2454] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774c258df5cc4adcb54b6f90ea831ad2 in term 0.
I20260927 18:50:05.627902 14254 raft_consensus.cc:504] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:05.627902 14255 raft_consensus.cc:504] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:05.628262 14255 leader_election.cc:288] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [CANDIDATE]: Term 1 election: Requested vote from peers 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761), 9a2d7e4a753e4994916fbeac465284b1 (127.29.159.65:39989)
I20260927 18:50:05.628289 14254 leader_election.cc:288] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [CANDIDATE]: Term 1 election: Requested vote from peers 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967), 9a2d7e4a753e4994916fbeac465284b1 (127.29.159.65:39989)
I20260927 18:50:05.628618 14191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7159cac07c9445296a2a5f95f171a81" candidate_uuid: "774c258df5cc4adcb54b6f90ea831ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da"
I20260927 18:50:05.628625 13921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7159cac07c9445296a2a5f95f171a81" candidate_uuid: "774c258df5cc4adcb54b6f90ea831ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2d7e4a753e4994916fbeac465284b1"
I20260927 18:50:05.628623 14059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7159cac07c9445296a2a5f95f171a81" candidate_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774c258df5cc4adcb54b6f90ea831ad2"
I20260927 18:50:05.628661 13920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7159cac07c9445296a2a5f95f171a81" candidate_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2d7e4a753e4994916fbeac465284b1"
I20260927 18:50:05.628778 14191 raft_consensus.cc:2379] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 774c258df5cc4adcb54b6f90ea831ad2 in current term 1: Already voted for candidate 2e3f3fed48f94c129dcc9f9ae1adf1da in this term.
I20260927 18:50:05.628779 14059 raft_consensus.cc:2379] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2e3f3fed48f94c129dcc9f9ae1adf1da in current term 1: Already voted for candidate 774c258df5cc4adcb54b6f90ea831ad2 in this term.
I20260927 18:50:05.628724 13921 raft_consensus.cc:3037] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:05.629232 14127 leader_election.cc:302] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e3f3fed48f94c129dcc9f9ae1adf1da; no voters: 774c258df5cc4adcb54b6f90ea831ad2, 9a2d7e4a753e4994916fbeac465284b1
I20260927 18:50:05.629416 14255 raft_consensus.cc:2726] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 18:50:05.629916 13921 raft_consensus.cc:2454] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774c258df5cc4adcb54b6f90ea831ad2 in term 1.
I20260927 18:50:05.630193 13995 leader_election.cc:302] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 774c258df5cc4adcb54b6f90ea831ad2, 9a2d7e4a753e4994916fbeac465284b1; no voters: 2e3f3fed48f94c129dcc9f9ae1adf1da
I20260927 18:50:05.630375 14254 raft_consensus.cc:2781] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:05.630564 14254 raft_consensus.cc:686] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 1 LEADER]: Becoming Leader. State: Replica: 774c258df5cc4adcb54b6f90ea831ad2, State: Running, Role: LEADER
I20260927 18:50:05.630891 14254 consensus_queue.cc:237] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:05.634068 13779 catalog_manager.cc:5495] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66). New cstate: current_term: 1 leader_uuid: "774c258df5cc4adcb54b6f90ea831ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } health_report { overall_health: UNKNOWN } } }
I20260927 18:50:05.647665 14248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.644111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.29.159.66:43761)
0927 18:50:05.644229 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:05.644236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:05.644377 (+ 141us) server_negotiation.cc:407] Connection header received
0927 18:50:05.644570 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:05.644572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:05.644620 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:05.644723 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:05.644954 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.645442 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.646263 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:05.646484 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:05.646595 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:05.646893 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:05.647268 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:05.647394 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:05.647483 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 18:50:05.647532 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:50:05.652930 13921 raft_consensus.cc:1261] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Refusing update from remote peer 774c258df5cc4adcb54b6f90ea831ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:05.652938 14191 raft_consensus.cc:1261] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 1 FOLLOWER]: Refusing update from remote peer 774c258df5cc4adcb54b6f90ea831ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:05.653505 14254 consensus_queue.cc:1035] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:05.653726 14260 consensus_queue.cc:1035] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:05.661505 14264 mvcc.cc:205] Tried to move back new op lower bound from 7334031383147958272 to 7334031383066861568. Current Snapshot: MvccSnapshot[applied={T|T < 7334031383147958272}]
I20260927 18:50:05.667659 13778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:05.667835 13778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:05.669948 13778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Sending DeleteTablet for 3 replicas of tablet e7159cac07c9445296a2a5f95f171a81
I20260927 18:50:05.670614 13901 tablet_service.cc:1514] Processing DeleteTablet for tablet e7159cac07c9445296a2a5f95f171a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:05 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:42726
I20260927 18:50:05.670729 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 13817
I20260927 18:50:05.671162 14171 tablet_service.cc:1514] Processing DeleteTablet for tablet e7159cac07c9445296a2a5f95f171a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:05 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:50556
I20260927 18:50:05.671657 14039 tablet_service.cc:1514] Processing DeleteTablet for tablet e7159cac07c9445296a2a5f95f171a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:05 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:39798
I20260927 18:50:05.672118 14272 tablet_replica.cc:330] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2: stopping tablet replica
I20260927 18:50:05.672371 14273 tablet_replica.cc:330] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da: stopping tablet replica
I20260927 18:50:05.672492 14272 raft_consensus.cc:2229] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:50:05.672655 14273 raft_consensus.cc:2229] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:05.672905 14272 raft_consensus.cc:2258] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:05.672966 14273 raft_consensus.cc:2258] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:05.674381 14273 ts_tablet_manager.cc:1905] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:05.674381 14272 ts_tablet_manager.cc:1905] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:05.677049 14272 ts_tablet_manager.cc:1918] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:05.677050 14273 ts_tablet_manager.cc:1918] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:05.677184 14272 log.cc:1196] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/e7159cac07c9445296a2a5f95f171a81
I20260927 18:50:05.677191 14273 log.cc:1196] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/e7159cac07c9445296a2a5f95f171a81
I20260927 18:50:05.677575 14273 ts_tablet_manager.cc:1939] T e7159cac07c9445296a2a5f95f171a81 P 2e3f3fed48f94c129dcc9f9ae1adf1da: Deleting consensus metadata
I20260927 18:50:05.677589 14272 ts_tablet_manager.cc:1939] T e7159cac07c9445296a2a5f95f171a81 P 774c258df5cc4adcb54b6f90ea831ad2: Deleting consensus metadata
I20260927 18:50:05.678755 13763 catalog_manager.cc:4841] TS 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967): tablet e7159cac07c9445296a2a5f95f171a81 (table test-table [id=10b6347e0b7a42acabe0f0ceec0901cf]) successfully deleted
I20260927 18:50:05.678965 13763 catalog_manager.cc:4841] TS 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761): tablet e7159cac07c9445296a2a5f95f171a81 (table test-table [id=10b6347e0b7a42acabe0f0ceec0901cf]) successfully deleted
W20260927 18:50:05.679128 13766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:39989 (error 108)
I20260927 18:50:05.680194 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:05.679874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:39989 (local address 127.0.0.1:42738)
0927 18:50:05.680045 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:05.680114 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260927 18:50:05.680370 13766 catalog_manager.cc:4584] TS 9a2d7e4a753e4994916fbeac465284b1 (127.29.159.65:39989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7159cac07c9445296a2a5f95f171a81: Network error: Client connection negotiation failed: client connection to 127.29.159.65:39989: connect: Connection refused (error 111)
I20260927 18:50:05.681393 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 13972
I20260927 18:50:05.686808 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 14108
I20260927 18:50:05.692606 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 13746
I20260927 18:50:25.699177 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:36825
--webserver_interface=127.29.159.126
--webserver_port=43859
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:25.807914 14275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:25.808147 14275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:25.808185 14275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:25.812034 14275 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:50:25.812079 14275 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:50:25.812095 14275 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:50:25.812108 14275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:25.812126 14275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:25.812143 14275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:25.812160 14275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:25.812173 14275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:25.817240 14275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=43859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:25.818193 14275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:25.819134 14275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:25.827555 14280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:25.827611 14275 server_base.cc:1034] running on GCE node
W20260927 18:50:25.827540 14281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:25.827574 14283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:25.828298 14275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:25.829414 14275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:25.830603 14275 hybrid_clock.cc:648] HybridClock initialized: now 1790535025830589 us; error 26 us; skew 500 ppm
Sep 27 18:50:25 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535025, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:25.833432 14275 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:50:25.834275 14275 webserver.cc:460] Webserver started at http://127.29.159.126:43859/ using document root <none> and password file <none>
I20260927 18:50:25.834828 14275 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:25.834926 14275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:25.838600 14275 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:50:25.840642 14290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:25.841933 14275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:50:25.842084 14275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "19d12cf4bd2f44b9af310e4fc6081192"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "916ce4dab80e6d5b5048ab8833ecc09e"
server_key_iv: "bf5071fab93259963770c0540e133625"
server_key_version: "encryptionkey@0"
I20260927 18:50:25.842617 14275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:25.871945 14275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:25.875689 14275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:25.875896 14275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:25.882838 14275 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:36825
I20260927 18:50:25.883118 14342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:36825 every 8 connection(s)
I20260927 18:50:25.883996 14275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:25.885615 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 14275
I20260927 18:50:25.886039 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:39989
--local_ip_for_outbound_sockets=127.29.159.65
--tserver_master_addrs=127.29.159.126:36825
--webserver_port=36135
--webserver_interface=127.29.159.65
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:50:25.888329 14343 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:50:25.889922 14343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Bootstrap starting.
I20260927 18:50:25.906924 14343 log.cc:824] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:25.918069 14343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:50:25.918486 14343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: Bootstrap complete.
I20260927 18:50:25.922495 14343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:50:25.922895 14343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d12cf4bd2f44b9af310e4fc6081192, State: Initialized, Role: FOLLOWER
I20260927 18:50:25.923468 14343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:50:25.923626 14343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:25.923694 14343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:25.923786 14343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:50:25.925596 14343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:50:25.925930 14343 leader_election.cc:302] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d12cf4bd2f44b9af310e4fc6081192; no voters:
I20260927 18:50:25.926273 14343 leader_election.cc:288] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:50:25.926355 14347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:50:25.926587 14347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [term 3 LEADER]: Becoming Leader. State: Replica: 19d12cf4bd2f44b9af310e4fc6081192, State: Running, Role: LEADER
I20260927 18:50:25.926918 14347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } }
I20260927 18:50:25.927523 14343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:25.928496 14349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d12cf4bd2f44b9af310e4fc6081192. Latest consensus state: current_term: 3 leader_uuid: "19d12cf4bd2f44b9af310e4fc6081192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } } }
I20260927 18:50:25.928630 14349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:25.928558 14348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "19d12cf4bd2f44b9af310e4fc6081192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d12cf4bd2f44b9af310e4fc6081192" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 36825 } } }
I20260927 18:50:25.928700 14348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:25.933032 14362 catalog_manager.cc:1259] Loaded cluster ID: bbaecae441114c799b264ce688cdcb7f
I20260927 18:50:25.933101 14362 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: loading cluster ID for follower catalog manager: success
I20260927 18:50:25.935637 14362 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19d12cf4bd2f44b9af310e4fc6081192: acquiring CA information for follower catalog manager: success
F20260927 18:50:25.936815 14362 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5c14f409ad google::LogMessage::Fail() at ??:0
@ 0x7f5c14f4388a google::LogMessage::SendToLog() at ??:0
@ 0x7f5c14f404de google::LogMessage::Flush() at ??:0
@ 0x7f5c14f41a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5c17c0418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5c1f8a5989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f5c1f882023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f5c1f88254b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f5c1f87b652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f5c1f87a9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f5c1f8c1541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5c17183512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5c187c76db start_thread at ??:0
@ 0x7f5c13a3e61f clone at ??:0
W20260927 18:50:26.025564 14345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:26.025815 14345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:26.025857 14345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:26.029878 14345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:26.029933 14345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:26.030021 14345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:50:26.035149 14345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:39989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=36135
--tserver_master_addrs=127.29.159.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:26.036230 14345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:26.037240 14345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:26.046949 14383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:26.046923 14384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:26.047118 14345 server_base.cc:1034] running on GCE node
W20260927 18:50:26.046916 14386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:26.047775 14345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:26.048444 14345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:26.049654 14345 hybrid_clock.cc:648] HybridClock initialized: now 1790535026049642 us; error 30 us; skew 500 ppm
Sep 27 18:50:26 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535026, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:26.052531 14345 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:50:26.053366 14345 webserver.cc:460] Webserver started at http://127.29.159.65:36135/ using document root <none> and password file <none>
I20260927 18:50:26.053912 14345 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:26.053973 14345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:26.057929 14345 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:50:26.060333 14393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:26.061738 14345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:50:26.061915 14345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "9a2d7e4a753e4994916fbeac465284b1"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "ae59320a9425e71f0fd719e788760dfd"
server_key_iv: "0b691b59725f5173ddf541f4f3a77fbf"
server_key_version: "encryptionkey@0"
I20260927 18:50:26.062525 14345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:26.089808 14345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:26.093736 14345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:26.093971 14345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:26.094633 14345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:26.095894 14400 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 18:50:26.101389 14345 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:50:26.101492 14345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 18:50:26.101615 14345 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:50:26.103600 14345 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:50:26.103672 14345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 18:50:26.103951 14400 tablet_bootstrap.cc:492] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: Bootstrap starting.
I20260927 18:50:26.115331 14345 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:39989
I20260927 18:50:26.115612 14507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:39989 every 8 connection(s)
I20260927 18:50:26.116520 14345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:50:26.123133 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 14345
I20260927 18:50:26.123728 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:43761
--local_ip_for_outbound_sockets=127.29.159.66
--tserver_master_addrs=127.29.159.126:36825
--webserver_port=34745
--webserver_interface=127.29.159.66
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:50:26.125703 14400 log.cc:824] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:26.133205 14400 tablet_bootstrap.cc:492] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:50:26.133659 14400 tablet_bootstrap.cc:492] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: Bootstrap complete.
I20260927 18:50:26.134756 14400 ts_tablet_manager.cc:1397] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.009s
I20260927 18:50:26.137228 14400 raft_consensus.cc:348] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:26.137670 14400 raft_consensus.cc:729] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2d7e4a753e4994916fbeac465284b1, State: Initialized, Role: FOLLOWER
I20260927 18:50:26.138193 14400 consensus_queue.cc:260] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:26.139201 14400 ts_tablet_manager.cc:1428] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1: Time spent starting tablet: real 0.004s user 0.003s sys 0.004s
W20260927 18:50:26.153616 14510 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:50:26.118747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:44631)
0927 18:50:26.119277 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:26.119340 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:26.120308 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:26.152802 (+ 32494us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":147,"threads_started":1}
W20260927 18:50:26.154282 14508 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:36825 (0 consecutive failures): Network error: Failed to ping master at 127.29.159.126:36825: Client connection negotiation failed: client connection to 127.29.159.126:36825: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:50:26.155081 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:26.154821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:49941)
0927 18:50:26.154950 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:26.155027 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 18:50:26.155890 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:26.155692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:60257)
0927 18:50:26.155801 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:26.155844 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 18:50:26.156144 14508 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:50:26.250615 14511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:26.250881 14511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:26.250983 14511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:26.255064 14511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:26.255118 14511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:26.255204 14511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:50:26.260548 14511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:43761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=34745
--tserver_master_addrs=127.29.159.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:26.261556 14511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:26.262571 14511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:26.272372 14521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:26.272486 14519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:26.272393 14518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:26.272571 14511 server_base.cc:1034] running on GCE node
I20260927 18:50:26.273056 14511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:26.273670 14511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:26.274878 14511 hybrid_clock.cc:648] HybridClock initialized: now 1790535026274840 us; error 49 us; skew 500 ppm
Sep 27 18:50:26 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535026, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:26.277640 14511 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:50:26.278548 14511 webserver.cc:460] Webserver started at http://127.29.159.66:34745/ using document root <none> and password file <none>
I20260927 18:50:26.279105 14511 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:26.279165 14511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:26.282974 14511 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:26.285059 14528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:26.286237 14511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:50:26.286360 14511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "774c258df5cc4adcb54b6f90ea831ad2"
format_stamp: "Formatted at 2026-09-27 18:49:43 on dist-test-slave-8tg2"
server_key: "55893a9494a760e6f5e0463ce3c884dd"
server_key_iv: "b8734fc0fac89426312c4ef25dc2145a"
server_key_version: "encryptionkey@0"
I20260927 18:50:26.286875 14511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:26.297827 14511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:26.300578 14511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:26.300762 14511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:26.301366 14511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:26.302135 14511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:26.302181 14511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:26.302241 14511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:26.302273 14511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:26.310998 14511 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:43761
I20260927 18:50:26.311079 14641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:43761 every 8 connection(s)
I20260927 18:50:26.312176 14511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:50:26.315932 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:26.314203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:45103)
0927 18:50:26.315073 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:26.315197 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":539,"threads_started":1}
W20260927 18:50:26.316399 14642 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:36825 (0 consecutive failures): Network error: Failed to ping master at 127.29.159.126:36825: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
I20260927 18:50:26.316919 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:26.316726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:38593)
0927 18:50:26.316823 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:26.316869 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 18:50:26.317582 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:26.317305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:47679)
0927 18:50:26.317495 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:26.317534 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 18:50:26.317751 14642 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:50:26.321491 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 14511
I20260927 18:50:26.321909 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:41967
--local_ip_for_outbound_sockets=127.29.159.67
--tserver_master_addrs=127.29.159.126:36825
--webserver_port=44371
--webserver_interface=127.29.159.67
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:26.433753 14645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:26.434026 14645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:26.434070 14645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:26.438177 14645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:26.438231 14645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:26.438309 14645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:50:26.443476 14645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:41967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=44371
--tserver_master_addrs=127.29.159.126:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:26.444450 14645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:26.445456 14645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:26.454964 14650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:26.455005 14651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:26.454945 14653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:26.455392 14645 server_base.cc:1034] running on GCE node
I20260927 18:50:26.455811 14645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:26.456436 14645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:26.457623 14645 hybrid_clock.cc:648] HybridClock initialized: now 1790535026457580 us; error 52 us; skew 500 ppm
Sep 27 18:50:26 dist-test-slave-8tg2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535026, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:26.460424 14645 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:50:26.461328 14645 webserver.cc:460] Webserver started at http://127.29.159.67:44371/ using document root <none> and password file <none>
I20260927 18:50:26.461894 14645 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:26.462004 14645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:26.465971 14645 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:26.468335 14660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:26.469621 14645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:50:26.469715 14645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da"
format_stamp: "Formatted at 2026-09-27 18:49:44 on dist-test-slave-8tg2"
server_key: "a61b878dc5bc83d9635e998a5d91334f"
server_key_iv: "fc442182d4c8b71213c7d4d059cca5be"
server_key_version: "encryptionkey@0"
I20260927 18:50:26.470268 14645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:26.495014 14645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:26.498355 14645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:26.498555 14645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:26.499197 14645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:26.500043 14645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:26.500088 14645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:26.500157 14645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:26.500200 14645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:26.509553 14645 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:41967
I20260927 18:50:26.509635 14773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:41967 every 8 connection(s)
I20260927 18:50:26.510643 14645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:50:26.514621 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:26.512980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:34695)
0927 18:50:26.513452 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:26.513891 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
W20260927 18:50:26.515156 14774 heartbeater.cc:644] Failed to heartbeat to 127.29.159.126:36825 (0 consecutive failures): Network error: Failed to ping master at 127.29.159.126:36825: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
I20260927 18:50:26.515941 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:26.515645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:50165)
0927 18:50:26.515809 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:26.515877 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 18:50:26.516620 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:26.516439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:43503)
0927 18:50:26.516534 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:26.516576 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 18:50:26.516841 14774 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:50:26.518757 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 14645
I20260927 18:50:27.157267 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:27.156831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:60981)
0927 18:50:27.157099 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:27.157157 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260927 18:50:27.318655 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:27.318248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:42229)
0927 18:50:27.318506 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:27.318557 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":87,"threads_started":1}
I20260927 18:50:27.517786 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:27.517413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:46713)
0927 18:50:27.517631 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:27.517683 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
I20260927 18:50:27.998474 14781 raft_consensus.cc:482] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:27.998718 14781 raft_consensus.cc:504] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:27.999943 14781 leader_election.cc:288] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967), 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761)
Sep 27 18:50:28 dist-test-slave-8tg2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535026, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.66@KRBTEST.COM
Sep 27 18:50:28 dist-test-slave-8tg2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535026, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.67@KRBTEST.COM
I20260927 18:50:28.019586 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:28.000193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:41967 (local address 127.29.159.65:48143)
0927 18:50:28.000438 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:28.000454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:28.000557 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:28.002493 (+ 1936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:28.002498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:28.003024 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:28.003533 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:28.003551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.004789 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.004792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:28.005699 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:28.005708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.006438 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.006442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:28.006678 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:28.011943 (+ 5265us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:28.011966 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:28.014097 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:28.016775 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:28.016782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:28.016804 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:28.017322 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:28.017696 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:28.017699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:28.017700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:28.017855 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:28.018568 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:28.018574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:28.018844 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 18:50:28.018997 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 18:50:28.019355 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":137,"spinlock_wait_cycles":36096,"thread_start_us":86,"threads_started":1}
I20260927 18:50:28.019631 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:28.000442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:48143 (local address 127.29.159.67:41967)
0927 18:50:28.000831 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:28.000853 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:28.000892 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:50:28.001553 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:28.001558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:28.002035 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:28.002440 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:28.003696 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.004646 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.005823 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.006290 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.006487 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:28.014292 (+ 7805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:28.014299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:28.014316 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:28.014349 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:28.016636 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:28.017441 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:28.017448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:28.017454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:28.017596 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:28.017997 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:28.018002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:28.018004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:28.018445 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:28.018549 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:28.019140 (+ 591us) server_negotiation.cc:299] Negotiation successful
0927 18:50:28.019432 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":77,"threads_started":1}
I20260927 18:50:28.019639 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:28.000755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:40035 (local address 127.29.159.66:43761)
0927 18:50:28.001130 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:28.001140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:28.001168 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:50:28.001922 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:28.001930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:28.002287 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:28.002560 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:28.003704 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.004645 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.005830 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.006298 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.006480 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:28.014296 (+ 7816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:28.014301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:28.014316 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:28.014349 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:28.016660 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:28.017441 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:28.017455 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:28.017459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:28.017596 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:28.018091 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:28.018095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:28.018097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:28.018573 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:28.018703 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:28.019183 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 18:50:28.019471 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":89,"threads_started":1}
I20260927 18:50:28.019631 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:28.000579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:43761 (local address 127.29.159.65:40035)
0927 18:50:28.000815 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:28.000829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:28.000937 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:28.002501 (+ 1564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:28.002504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:28.003038 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:28.003534 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:28.003549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.004783 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.004786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:28.005696 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:28.005705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:28.006424 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:28.006428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:28.006680 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:28.011938 (+ 5258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:28.011957 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:28.013769 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:28.016775 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:28.016782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:28.016799 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:28.017322 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:28.017941 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:28.017942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:28.017944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:28.018000 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:28.018682 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:28.018684 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:28.018842 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 18:50:28.019038 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 18:50:28.019465 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":283,"spinlock_wait_cycles":36992,"thread_start_us":66,"threads_started":1}
W20260927 18:50:28.022567 14394 leader_election.cc:341] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761): Not found: Tablet not found: e7159cac07c9445296a2a5f95f171a81
W20260927 18:50:28.023276 14394 leader_election.cc:341] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967): Not found: Tablet not found: e7159cac07c9445296a2a5f95f171a81
I20260927 18:50:28.023378 14394 leader_election.cc:302] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a2d7e4a753e4994916fbeac465284b1; no voters: 2e3f3fed48f94c129dcc9f9ae1adf1da, 774c258df5cc4adcb54b6f90ea831ad2
I20260927 18:50:28.023691 14781 raft_consensus.cc:2726] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:50:28.158327 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:28.158040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:39745)
0927 18:50:28.158169 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:28.158229 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 18:50:28.319854 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:28.319365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:48735)
0927 18:50:28.319697 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:28.319750 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":166,"threads_started":1}
I20260927 18:50:28.518914 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:28.518507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:50655)
0927 18:50:28.518759 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:28.518814 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260927 18:50:29.159550 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:29.159071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:54845)
0927 18:50:29.159317 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:29.159374 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260927 18:50:29.321071 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:29.320646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:54887)
0927 18:50:29.320906 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:29.320971 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260927 18:50:29.520273 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:29.519777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:38319)
0927 18:50:29.520111 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:29.520176 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":87,"threads_started":1}
I20260927 18:50:30.160758 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:30.160273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:46515)
0927 18:50:30.160575 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:30.160641 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":110,"threads_started":1}
I20260927 18:50:30.210677 14792 raft_consensus.cc:482] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:30.210805 14792 raft_consensus.cc:504] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:30.211242 14792 leader_election.cc:288] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967), 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761)
W20260927 18:50:30.212117 14394 leader_election.cc:341] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967): Not found: Tablet not found: e7159cac07c9445296a2a5f95f171a81
W20260927 18:50:30.212236 14394 leader_election.cc:341] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761): Not found: Tablet not found: e7159cac07c9445296a2a5f95f171a81
I20260927 18:50:30.212283 14394 leader_election.cc:302] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a2d7e4a753e4994916fbeac465284b1; no voters: 2e3f3fed48f94c129dcc9f9ae1adf1da, 774c258df5cc4adcb54b6f90ea831ad2
I20260927 18:50:30.212423 14792 raft_consensus.cc:2726] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:50:30.322304 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:30.321863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:37043)
0927 18:50:30.322135 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:30.322193 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260927 18:50:30.521343 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:30.520973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:40999)
0927 18:50:30.521198 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:30.521250 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260927 18:50:31.162559 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:31.162044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:39443)
0927 18:50:31.162367 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:31.162429 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
I20260927 18:50:31.323698 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:31.323124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:55597)
0927 18:50:31.323375 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:31.323591 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260927 18:50:31.522576 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:31.522109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:33711)
0927 18:50:31.522425 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:31.522483 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
I20260927 18:50:32.164149 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:32.163751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:55845)
0927 18:50:32.163992 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:32.164052 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260927 18:50:32.324846 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:32.324423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:33243)
0927 18:50:32.324691 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:32.324749 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":93,"threads_started":1}
I20260927 18:50:32.523882 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:32.523388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:46019)
0927 18:50:32.523711 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:32.523774 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":153,"threads_started":1}
I20260927 18:50:32.981438 14801 raft_consensus.cc:482] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:32.981570 14801 raft_consensus.cc:504] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:32.982054 14801 leader_election.cc:288] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967), 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761)
W20260927 18:50:32.982856 14394 leader_election.cc:341] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967): Not found: Tablet not found: e7159cac07c9445296a2a5f95f171a81
W20260927 18:50:32.982986 14394 leader_election.cc:341] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761): Not found: Tablet not found: e7159cac07c9445296a2a5f95f171a81
I20260927 18:50:32.983062 14394 leader_election.cc:302] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a2d7e4a753e4994916fbeac465284b1; no voters: 2e3f3fed48f94c129dcc9f9ae1adf1da, 774c258df5cc4adcb54b6f90ea831ad2
I20260927 18:50:32.983239 14801 raft_consensus.cc:2726] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:50:33.165287 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:33.164906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:45059)
0927 18:50:33.165140 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:33.165193 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260927 18:50:33.326057 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:33.325617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:39757)
0927 18:50:33.325923 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:33.325973 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":69,"threads_started":1}
I20260927 18:50:33.525081 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:33.524652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:35031)
0927 18:50:33.524932 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:33.524989 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":68,"threads_started":1}
I20260927 18:50:34.166450 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.166055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:60233)
0927 18:50:34.166298 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.166357 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260927 18:50:34.327240 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.326834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:39425)
0927 18:50:34.327097 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.327150 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":101,"threads_started":1}
I20260927 18:50:34.526473 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:34.525997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:36841)
0927 18:50:34.526285 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:34.526353 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260927 18:50:35.167750 14808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.167265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:33911)
0927 18:50:35.167578 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.167656 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":93,"threads_started":1}
I20260927 18:50:35.328543 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.328119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:49335)
0927 18:50:35.328375 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.328432 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
I20260927 18:50:35.527738 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:35.527267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:55175)
0927 18:50:35.527578 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:35.527638 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260927 18:50:36.169056 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:36.168659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:55427)
0927 18:50:36.168898 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:36.168963 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260927 18:50:36.329916 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:36.329465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:50911)
0927 18:50:36.329752 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:36.329814 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":112,"threads_started":1}
I20260927 18:50:36.528983 14813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:36.528577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:48441)
0927 18:50:36.528832 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:36.528888 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260927 18:50:37.170204 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:37.169811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:48663)
0927 18:50:37.170058 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:37.170112 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260927 18:50:37.313704 14815 raft_consensus.cc:482] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:37.313829 14815 raft_consensus.cc:504] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:37.314268 14815 leader_election.cc:288] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967), 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761)
W20260927 18:50:37.315016 14394 leader_election.cc:341] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967): Not found: Tablet not found: e7159cac07c9445296a2a5f95f171a81
W20260927 18:50:37.315132 14394 leader_election.cc:341] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761): Not found: Tablet not found: e7159cac07c9445296a2a5f95f171a81
I20260927 18:50:37.315171 14394 leader_election.cc:302] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a2d7e4a753e4994916fbeac465284b1; no voters: 2e3f3fed48f94c129dcc9f9ae1adf1da, 774c258df5cc4adcb54b6f90ea831ad2
I20260927 18:50:37.315290 14815 raft_consensus.cc:2726] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:50:37.331339 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:37.330912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:51973)
0927 18:50:37.331174 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:37.331230 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
I20260927 18:50:37.530215 14817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:37.529809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:58775)
0927 18:50:37.530059 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:37.530118 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260927 18:50:38.171406 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:38.170946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:46293)
0927 18:50:38.171238 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:38.171302 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":74,"threads_started":1}
I20260927 18:50:38.332661 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:38.332205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:47571)
0927 18:50:38.332497 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:38.332563 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":74,"threads_started":1}
I20260927 18:50:38.531584 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:38.531127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:47673)
0927 18:50:38.531382 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:38.531481 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260927 18:50:39.172646 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.172194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:33143)
0927 18:50:39.172478 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.172545 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260927 18:50:39.334062 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.333559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:40703)
0927 18:50:39.333885 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.333951 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
I20260927 18:50:39.532855 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:39.532407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:60605)
0927 18:50:39.532697 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:39.532756 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":111,"threads_started":1}
I20260927 18:50:40.173978 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.173445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:46955)
0927 18:50:40.173761 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.173833 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
I20260927 18:50:40.335511 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.334997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:48247)
0927 18:50:40.335270 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.335334 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20260927 18:50:40.491695 14826 raft_consensus.cc:482] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:40.491858 14826 raft_consensus.cc:504] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3f3fed48f94c129dcc9f9ae1adf1da" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41967 } } peers { permanent_uuid: "774c258df5cc4adcb54b6f90ea831ad2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 43761 } } peers { permanent_uuid: "9a2d7e4a753e4994916fbeac465284b1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 39989 } }
I20260927 18:50:40.492267 14826 leader_election.cc:288] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967), 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761)
W20260927 18:50:40.493093 14394 leader_election.cc:341] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 774c258df5cc4adcb54b6f90ea831ad2 (127.29.159.66:43761): Not found: Tablet not found: e7159cac07c9445296a2a5f95f171a81
W20260927 18:50:40.493252 14394 leader_election.cc:341] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e3f3fed48f94c129dcc9f9ae1adf1da (127.29.159.67:41967): Not found: Tablet not found: e7159cac07c9445296a2a5f95f171a81
I20260927 18:50:40.493319 14394 leader_election.cc:302] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a2d7e4a753e4994916fbeac465284b1; no voters: 2e3f3fed48f94c129dcc9f9ae1adf1da, 774c258df5cc4adcb54b6f90ea831ad2
I20260927 18:50:40.493541 14826 raft_consensus.cc:2726] T e7159cac07c9445296a2a5f95f171a81 P 9a2d7e4a753e4994916fbeac465284b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:50:40.534276 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:40.533697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.67:57423)
0927 18:50:40.534077 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:40.534147 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":115,"threads_started":1}
I20260927 18:50:41.175235 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:41.174853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.65:58389)
0927 18:50:41.175088 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:41.175140 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260927 18:50:41.337112 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:41.336635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:36825 (local address 127.29.159.66:45595)
0927 18:50:41.336918 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:41.336977 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.126:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
I20260927 18:50:41.522130 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 14345
I20260927 18:50:41.527992 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 14511
I20260927 18:50:41.533370 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 14645
2026-09-27T18:50:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59574 ms)
[----------] 30 tests from SecurityITest (240384 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:41 dist-test-slave-8tg2 krb5kdc[14832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:41 dist-test-slave-8tg2 krb5kdc[14832](info): set up 2 sockets
Sep 27 18:50:41 dist-test-slave-8tg2 krb5kdc[14832](info): commencing operation
krb5kdc: starting...
W20260927 18:50:42.603693 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:50:42 dist-test-slave-8tg2 krb5kdc[14832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:50:42.730587 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44733
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:44733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:42.841431 14845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:42.841711 14845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:42.841786 14845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:42.845844 14845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:42.845901 14845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:42.845919 14845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:42.845965 14845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:42.845996 14845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:42.851272 14845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:44733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:42.852411 14845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:42.853484 14845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:50:42.862702 14845 server_base.cc:1034] running on GCE node
W20260927 18:50:42.862738 14853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:42.862891 14850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:42.862738 14851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:42.863641 14845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:42.864779 14845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:42.865981 14845 hybrid_clock.cc:648] HybridClock initialized: now 1790535042865960 us; error 34 us; skew 500 ppm
Sep 27 18:50:42 dist-test-slave-8tg2 krb5kdc[14832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535042, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:42.868955 14845 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:50:42.869903 14845 webserver.cc:460] Webserver started at http://127.29.159.126:35545/ using document root <none> and password file <none>
I20260927 18:50:42.870447 14845 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:42.870508 14845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:42.870676 14845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:42.872785 14845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "fc14a945fc824af199b4ece821bc9a75"
format_stamp: "Formatted at 2026-09-27 18:50:42 on dist-test-slave-8tg2"
server_key: "4593d3134a8d5ece451e4578c6485263"
server_key_iv: "2d640fc4e39d79c78b0ded4f2cedf813"
server_key_version: "encryptionkey@0"
I20260927 18:50:42.873302 14845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "fc14a945fc824af199b4ece821bc9a75"
format_stamp: "Formatted at 2026-09-27 18:50:42 on dist-test-slave-8tg2"
server_key: "4593d3134a8d5ece451e4578c6485263"
server_key_iv: "2d640fc4e39d79c78b0ded4f2cedf813"
server_key_version: "encryptionkey@0"
I20260927 18:50:42.877031 14845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:50:42.879354 14860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:42.880646 14845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:42.880779 14845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "fc14a945fc824af199b4ece821bc9a75"
format_stamp: "Formatted at 2026-09-27 18:50:42 on dist-test-slave-8tg2"
server_key: "4593d3134a8d5ece451e4578c6485263"
server_key_iv: "2d640fc4e39d79c78b0ded4f2cedf813"
server_key_version: "encryptionkey@0"
I20260927 18:50:42.880901 14845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:42.898656 14845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:42.902096 14845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:42.902314 14845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:42.909988 14845 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:44733
I20260927 18:50:42.910006 14912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:44733 every 8 connection(s)
I20260927 18:50:42.910989 14845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:42.913880 14913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:42.916899 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 14845
I20260927 18:50:42.917011 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:50:42.917315 30333 external_mini_cluster.cc:1428] Setting key 6fb9f93960a774e46f346f52ec627849
I20260927 18:50:42.920044 14913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75: Bootstrap starting.
I20260927 18:50:42.922819 14913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:42.923712 14913 log.cc:824] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:42.925879 14913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75: No bootstrap required, opened a new log
Sep 27 18:50:42 dist-test-slave-8tg2 krb5kdc[14832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:42.929898 14913 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc14a945fc824af199b4ece821bc9a75" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44733 } }
I20260927 18:50:42.930204 14913 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:42.930277 14913 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc14a945fc824af199b4ece821bc9a75, State: Initialized, Role: FOLLOWER
I20260927 18:50:42.930795 14913 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc14a945fc824af199b4ece821bc9a75" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44733 } }
I20260927 18:50:42.930958 14913 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:42.931034 14913 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:42.931159 14913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:42.932477 14913 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc14a945fc824af199b4ece821bc9a75" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44733 } }
I20260927 18:50:42.932857 14913 leader_election.cc:302] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc14a945fc824af199b4ece821bc9a75; no voters:
I20260927 18:50:42.933209 14913 leader_election.cc:288] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:42.933334 14918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:42.933593 14918 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [term 1 LEADER]: Becoming Leader. State: Replica: fc14a945fc824af199b4ece821bc9a75, State: Running, Role: LEADER
I20260927 18:50:42.933954 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:42.918838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.29.159.126:44733)
0927 18:50:42.919277 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:42.919296 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:42.919340 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:50:42.920201 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:42.920224 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:42.920642 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:42.921020 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:42.921701 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:42.923294 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:42.924354 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:42.925027 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:42.925206 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:42.927790 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:42.927797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:42.927814 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:42.927847 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:42.930463 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:42.931003 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:42.931010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:42.931015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:42.931204 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:42.931535 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:42.931538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:42.931539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:42.932083 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:42.932283 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:42.932584 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 18:50:42.932904 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":119,"threads_started":1}
I20260927 18:50:42.934007 14918 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc14a945fc824af199b4ece821bc9a75" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44733 } }
I20260927 18:50:42.934583 14913 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:42.935576 14918 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc14a945fc824af199b4ece821bc9a75. Latest consensus state: current_term: 1 leader_uuid: "fc14a945fc824af199b4ece821bc9a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc14a945fc824af199b4ece821bc9a75" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44733 } } }
I20260927 18:50:42.936332 14918 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:42.936218 14919 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc14a945fc824af199b4ece821bc9a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc14a945fc824af199b4ece821bc9a75" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44733 } } }
I20260927 18:50:42.936657 14919 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75 [sys.catalog]: This master's current role is: LEADER
W20260927 18:50:42.939347 14933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:50:42.939481 14933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:50:42.939592 14932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:42.940315 14932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:42.946050 14932 catalog_manager.cc:1347] Generated new cluster ID: 64fd3f1b381e410cb2edec25897e6e9c
I20260927 18:50:42.946275 14932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:42.965287 14932 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:42.966496 14932 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:42.970914 14932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75: Generated new TSK 0
I20260927 18:50:42.971666 14932 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:50:43.003350 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44733
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:43.137639 14939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:43.137918 14939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:43.137965 14939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:43.142135 14939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:43.142198 14939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:43.142257 14939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:50:43.147516 14939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:43.148528 14939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:43.149519 14939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:43.159227 14944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:43.159222 14947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:43.159377 14939 server_base.cc:1034] running on GCE node
W20260927 18:50:43.159222 14945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:43.159919 14939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:43.160602 14939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:43.161791 14939 hybrid_clock.cc:648] HybridClock initialized: now 1790535043161774 us; error 29 us; skew 500 ppm
Sep 27 18:50:43 dist-test-slave-8tg2 krb5kdc[14832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535043, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:43.164685 14939 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:50:43.165620 14939 webserver.cc:460] Webserver started at http://127.29.159.65:33901/ using document root <none> and password file <none>
I20260927 18:50:43.166201 14939 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:43.166293 14939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:43.166534 14939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:43.168684 14939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "7bbd0f3b2fb64786b2b691e79ac3be18"
format_stamp: "Formatted at 2026-09-27 18:50:43 on dist-test-slave-8tg2"
server_key: "642590adb7bfc395a00f559ee8f57749"
server_key_iv: "81fefd0381815a35c4ff979249c689be"
server_key_version: "encryptionkey@0"
I20260927 18:50:43.169271 14939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "7bbd0f3b2fb64786b2b691e79ac3be18"
format_stamp: "Formatted at 2026-09-27 18:50:43 on dist-test-slave-8tg2"
server_key: "642590adb7bfc395a00f559ee8f57749"
server_key_iv: "81fefd0381815a35c4ff979249c689be"
server_key_version: "encryptionkey@0"
I20260927 18:50:43.173120 14939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 18:50:43.175704 14954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:43.176961 14939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:50:43.177095 14939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "7bbd0f3b2fb64786b2b691e79ac3be18"
format_stamp: "Formatted at 2026-09-27 18:50:43 on dist-test-slave-8tg2"
server_key: "642590adb7bfc395a00f559ee8f57749"
server_key_iv: "81fefd0381815a35c4ff979249c689be"
server_key_version: "encryptionkey@0"
I20260927 18:50:43.177220 14939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:43.201385 14939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:43.205042 14939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:43.205268 14939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:43.205930 14939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:43.207012 14939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:43.207079 14939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:43.207134 14939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:43.207163 14939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:43.216734 14939 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:33139
I20260927 18:50:43.216753 15067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:33139 every 8 connection(s)
I20260927 18:50:43.217654 14939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:50:43.221267 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 14939
I20260927 18:50:43.221403 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:50:43.221720 30333 external_mini_cluster.cc:1428] Setting key 4e0fba879d95e9bf8a257fb4c2df5d63
Sep 27 18:50:43 dist-test-slave-8tg2 krb5kdc[14832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535043, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:50:43.232416 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.219523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:34001 (local address 127.29.159.126:44733)
0927 18:50:43.219665 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:43.219669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:43.220502 (+ 833us) server_negotiation.cc:407] Connection header received
0927 18:50:43.222240 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:43.222244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:43.222302 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:43.222412 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:43.223747 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.224272 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.225614 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.225854 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.225973 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.228586 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:43.228588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:43.228591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:43.228615 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:43.230119 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:43.230664 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:43.230673 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:43.230674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:43.230784 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:43.231153 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:43.231156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:43.231158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:43.231347 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:43.231473 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:43.232104 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 18:50:43.232235 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
I20260927 18:50:43.233459 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.219875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44733 (local address 127.29.159.65:34001)
0927 18:50:43.220341 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:43.220383 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:43.221990 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:43.222597 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:43.222604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:43.222939 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:43.223593 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.223614 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.224403 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.224409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.225478 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.225487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.226011 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.226015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.226182 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.226867 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:43.226891 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:43.228388 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:43.230244 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:43.230251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:43.230268 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:43.230551 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:43.230883 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:43.230888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:43.230890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:43.231040 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:43.231483 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:43.231492 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:43.231859 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 18:50:43.232050 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 18:50:43.232390 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260927 18:50:43.235037 15068 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44733
I20260927 18:50:43.235383 15068 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:43.235976 15068 heartbeater.cc:511] Master 127.29.159.126:44733 requested a full tablet report, sending...
I20260927 18:50:43.237784 14877 ts_manager.cc:194] Registered new tserver with Master: 7bbd0f3b2fb64786b2b691e79ac3be18 (127.29.159.65:33139)
I20260927 18:50:43.239611 14877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:34001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:50:43.252368 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44733
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:43.361444 15073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:43.361701 15073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:43.361743 15073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:43.365607 15073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:43.365655 15073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:43.365710 15073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:50:43.370843 15073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:43.371834 15073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:43.372778 15073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:43.382216 15081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:43.382212 15078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:43.382231 15079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:43.382640 15073 server_base.cc:1034] running on GCE node
I20260927 18:50:43.383045 15073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:43.383760 15073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:43.384936 15073 hybrid_clock.cc:648] HybridClock initialized: now 1790535043384923 us; error 31 us; skew 500 ppm
Sep 27 18:50:43 dist-test-slave-8tg2 krb5kdc[14832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535043, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:43.387759 15073 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:50:43.388624 15073 webserver.cc:460] Webserver started at http://127.29.159.66:43279/ using document root <none> and password file <none>
I20260927 18:50:43.389183 15073 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:43.389284 15073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:43.389495 15073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:43.391564 15073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "284a2fd1811d40f09307078c1e5d6cfb"
format_stamp: "Formatted at 2026-09-27 18:50:43 on dist-test-slave-8tg2"
server_key: "89fc3537bd5b4fee7956d073c344c8b5"
server_key_iv: "5ee889c76b814fbc2f423b4874f1527c"
server_key_version: "encryptionkey@0"
I20260927 18:50:43.392122 15073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "284a2fd1811d40f09307078c1e5d6cfb"
format_stamp: "Formatted at 2026-09-27 18:50:43 on dist-test-slave-8tg2"
server_key: "89fc3537bd5b4fee7956d073c344c8b5"
server_key_iv: "5ee889c76b814fbc2f423b4874f1527c"
server_key_version: "encryptionkey@0"
I20260927 18:50:43.395709 15073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:43.398028 15088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:43.399230 15073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:50:43.399343 15073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "284a2fd1811d40f09307078c1e5d6cfb"
format_stamp: "Formatted at 2026-09-27 18:50:43 on dist-test-slave-8tg2"
server_key: "89fc3537bd5b4fee7956d073c344c8b5"
server_key_iv: "5ee889c76b814fbc2f423b4874f1527c"
server_key_version: "encryptionkey@0"
I20260927 18:50:43.399510 15073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:43.409325 15073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:43.412007 15073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:43.412196 15073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:43.412746 15073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:43.413513 15073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:43.413559 15073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:43.413599 15073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:43.413614 15073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:43.422463 15073 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:44247
I20260927 18:50:43.422492 15201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:44247 every 8 connection(s)
I20260927 18:50:43.423370 15073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:50:43.428748 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 15073
I20260927 18:50:43.428905 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:50:43.429217 30333 external_mini_cluster.cc:1428] Setting key a3d61f1d977165c4537cfa59e96ee29f
Sep 27 18:50:43 dist-test-slave-8tg2 krb5kdc[14832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535043, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:43.438482 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.425434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:56937 (local address 127.29.159.126:44733)
0927 18:50:43.425589 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:43.425594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:43.426510 (+ 916us) server_negotiation.cc:407] Connection header received
0927 18:50:43.427888 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:43.427891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:43.427948 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:43.428035 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:43.429542 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.430175 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.431419 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.431678 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.431791 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.434256 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:43.434259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:43.434261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:43.434284 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:43.436003 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:43.436523 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:43.436527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:43.436529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:43.436612 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:43.437073 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:43.437080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:43.437106 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:43.437403 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:43.437537 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:43.438085 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 18:50:43.438241 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:50:43.439235 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.425847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44733 (local address 127.29.159.66:56937)
0927 18:50:43.426344 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:43.426397 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:43.427667 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:43.428195 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:43.428202 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:43.428658 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:43.429392 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.429413 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.430303 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.430306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.431293 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.431303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.431829 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.431833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.432003 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.432632 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:43.432648 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:43.434066 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:43.436120 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:43.436126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:43.436142 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:43.436412 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:43.436704 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:43.436709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:43.436711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:43.436944 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:43.437546 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:43.437553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:43.437832 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 18:50:43.438010 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 18:50:43.438269 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
I20260927 18:50:43.440485 15202 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44733
I20260927 18:50:43.440711 15202 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:43.441213 15202 heartbeater.cc:511] Master 127.29.159.126:44733 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:50:43.442399 14877 ts_manager.cc:194] Registered new tserver with Master: 284a2fd1811d40f09307078c1e5d6cfb (127.29.159.66:44247)
I20260927 18:50:43.443260 14877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:56937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:50:43.459201 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44733
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:35407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:43.569466 15207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:43.569777 15207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:43.569849 15207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:43.573887 15207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:43.573978 15207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:43.574072 15207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:50:43.579414 15207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:43.580631 15207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:43.581773 15207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:43.591398 15213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:43.591509 15215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:43.591576 15212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:43.591770 15207 server_base.cc:1034] running on GCE node
I20260927 18:50:43.592221 15207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:43.592888 15207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:43.594074 15207 hybrid_clock.cc:648] HybridClock initialized: now 1790535043594036 us; error 49 us; skew 500 ppm
Sep 27 18:50:43 dist-test-slave-8tg2 krb5kdc[14832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535043, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:43.596741 15207 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:50:43.597604 15207 webserver.cc:460] Webserver started at http://127.29.159.67:45099/ using document root <none> and password file <none>
I20260927 18:50:43.598152 15207 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:43.598239 15207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:43.598483 15207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:43.600529 15207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "88a9b61365ff427686a96bd5c79da965"
format_stamp: "Formatted at 2026-09-27 18:50:43 on dist-test-slave-8tg2"
server_key: "9c19308dd452647d50fa96912c242166"
server_key_iv: "166e04f975dff64e5ae94799e98aa7a1"
server_key_version: "encryptionkey@0"
I20260927 18:50:43.601087 15207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "88a9b61365ff427686a96bd5c79da965"
format_stamp: "Formatted at 2026-09-27 18:50:43 on dist-test-slave-8tg2"
server_key: "9c19308dd452647d50fa96912c242166"
server_key_iv: "166e04f975dff64e5ae94799e98aa7a1"
server_key_version: "encryptionkey@0"
I20260927 18:50:43.604981 15207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:50:43.607285 15222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:43.608527 15207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:43.608657 15207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "88a9b61365ff427686a96bd5c79da965"
format_stamp: "Formatted at 2026-09-27 18:50:43 on dist-test-slave-8tg2"
server_key: "9c19308dd452647d50fa96912c242166"
server_key_iv: "166e04f975dff64e5ae94799e98aa7a1"
server_key_version: "encryptionkey@0"
I20260927 18:50:43.608784 15207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:43.628327 15207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:43.631474 15207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:43.631686 15207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:43.632292 15207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:43.633172 15207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:43.633240 15207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:43.633313 15207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:43.633366 15207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:43.642748 15207 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:42791
I20260927 18:50:43.642805 15335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:42791 every 8 connection(s)
I20260927 18:50:43.643731 15207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:50:43.644994 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 15207
I20260927 18:50:43.645150 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:50:43.645517 30333 external_mini_cluster.cc:1428] Setting key b6331aa7fe784e577ad0bcbb060e0b4c
Sep 27 18:50:43 dist-test-slave-8tg2 krb5kdc[14832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535043, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:43.658551 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.645628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:43249 (local address 127.29.159.126:44733)
0927 18:50:43.645762 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:43.645772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:43.646783 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 18:50:43.648362 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:43.648365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:43.648419 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:43.648495 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:43.649651 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.650198 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.651925 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.652151 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.652262 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.654753 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:43.654762 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:43.654764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:43.654787 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:43.656207 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:43.656733 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:43.656737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:43.656739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:43.656820 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:43.657175 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:43.657179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:43.657181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:43.657368 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:43.657477 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:43.658253 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 18:50:43.658390 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:50:43.659469 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.646073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44733 (local address 127.29.159.67:43249)
0927 18:50:43.646634 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:43.646692 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:43.648151 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:43.648669 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:43.648676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:43.649024 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:43.649511 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.649524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.650345 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.650352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.651794 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.651804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.652298 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.652302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.652493 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.653128 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:43.653144 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:43.654582 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:43.656326 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:43.656332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:43.656348 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:43.656625 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:43.656931 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:43.656934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:43.656936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:43.657078 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:43.657474 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:43.657482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:43.657747 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 18:50:43.657947 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 18:50:43.658335 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":158,"threads_started":1}
I20260927 18:50:43.660676 15336 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44733
I20260927 18:50:43.660902 15336 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:43.661361 15336 heartbeater.cc:511] Master 127.29.159.126:44733 requested a full tablet report, sending...
I20260927 18:50:43.662314 14877 ts_manager.cc:194] Registered new tserver with Master: 88a9b61365ff427686a96bd5c79da965 (127.29.159.67:42791)
I20260927 18:50:43.662937 14877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:43249
I20260927 18:50:43.672406 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:50:43.681941 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.674391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.29.159.126:44733)
0927 18:50:43.674535 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:43.674540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:43.674876 (+ 336us) server_negotiation.cc:407] Connection header received
0927 18:50:43.674989 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:43.674992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:43.675035 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:43.675144 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:43.675919 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.676445 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.677257 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.677473 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.677551 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.678839 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:43.678842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:43.678845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:43.678868 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:43.680242 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:43.680631 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:43.680634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:43.680635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:43.680712 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:43.681054 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:43.681057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:43.681058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:43.681282 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:43.681389 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:43.681622 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 18:50:43.681719 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:50:43.685132 14877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:50:43.688267 14877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:50:43.702724 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.696805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:42791 (local address 127.0.0.1:34642)
0927 18:50:43.697228 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:43.697274 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:43.697411 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:43.698284 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:43.698290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:43.698328 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:43.698512 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.698518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.699797 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.699800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.701088 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.701099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.702165 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.702168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.702270 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.702306 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:50:43.702446 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 18:50:43.702574 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":86,"thread_start_us":125,"threads_started":1}
I20260927 18:50:43.702731 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.696911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.29.159.67:42791)
0927 18:50:43.697586 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:43.697601 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:43.697627 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:50:43.697711 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:43.697716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:43.697927 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:43.698106 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:43.698607 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.699665 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.701219 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.702032 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.702156 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.702315 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:43.702484 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 18:50:43.702582 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":145,"threads_started":1}
I20260927 18:50:43.703258 15350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.696881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:44247 (local address 127.0.0.1:34656)
0927 18:50:43.698327 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:43.698341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:43.698432 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:43.698873 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:43.698876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:43.698896 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:43.698987 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.698993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.700459 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.700475 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.701565 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.701573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.702848 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.702867 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.702972 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.702994 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:50:43.703053 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 18:50:43.703120 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"mutex_wait_us":71,"thread_start_us":101,"threads_started":1}
I20260927 18:50:43.703558 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.697577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:33139 (local address 127.0.0.1:49938)
0927 18:50:43.698382 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:43.698414 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:43.698512 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:43.699116 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:43.699118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:43.699132 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:43.699194 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.699198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.700550 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.700553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.701652 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.701663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.703161 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.703164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.703243 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.703265 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:50:43.703330 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 18:50:43.703415 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260927 18:50:43.703979 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.697754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.29.159.65:33139)
0927 18:50:43.698180 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:43.698195 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:43.698603 (+ 408us) server_negotiation.cc:407] Connection header received
0927 18:50:43.698686 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:43.698691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:43.698850 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:43.698981 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:43.699306 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.700417 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.701784 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.702574 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.702667 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.703667 (+ 1000us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:43.703724 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 18:50:43.703829 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":136,"threads_started":1}
I20260927 18:50:43.704382 15352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.697861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.29.159.66:44247)
0927 18:50:43.698294 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:43.698321 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:43.698445 (+ 124us) server_negotiation.cc:407] Connection header received
0927 18:50:43.698535 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:43.698543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:43.698724 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:43.698865 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:43.699108 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.700297 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.701684 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.702553 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.702667 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.703500 (+ 833us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:43.703551 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 18:50:43.703649 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":138,"threads_started":1}
I20260927 18:50:43.706108 15270 tablet_service.cc:1467] Processing CreateTablet for tablet 8706301ec91b48c1881e221bfec2521b (DEFAULT_TABLE table=test-table [id=c6f03ff1bddf48f48f4f319c889effe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:43.707101 15270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8706301ec91b48c1881e221bfec2521b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:43.706898 15136 tablet_service.cc:1467] Processing CreateTablet for tablet 8706301ec91b48c1881e221bfec2521b (DEFAULT_TABLE table=test-table [id=c6f03ff1bddf48f48f4f319c889effe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:43.706898 15002 tablet_service.cc:1467] Processing CreateTablet for tablet 8706301ec91b48c1881e221bfec2521b (DEFAULT_TABLE table=test-table [id=c6f03ff1bddf48f48f4f319c889effe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:43.707970 15002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8706301ec91b48c1881e221bfec2521b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:43.707970 15136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8706301ec91b48c1881e221bfec2521b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:43.713651 15353 tablet_bootstrap.cc:492] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965: Bootstrap starting.
I20260927 18:50:43.715787 15355 tablet_bootstrap.cc:492] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18: Bootstrap starting.
I20260927 18:50:43.716096 15353 tablet_bootstrap.cc:654] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:43.717414 15353 log.cc:824] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:43.717921 15354 tablet_bootstrap.cc:492] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb: Bootstrap starting.
I20260927 18:50:43.718117 15355 tablet_bootstrap.cc:654] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:43.719331 15355 log.cc:824] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:43.719918 15353 tablet_bootstrap.cc:492] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965: No bootstrap required, opened a new log
I20260927 18:50:43.720124 15353 ts_tablet_manager.cc:1397] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:50:43.720193 15354 tablet_bootstrap.cc:654] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:43.721101 15355 tablet_bootstrap.cc:492] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18: No bootstrap required, opened a new log
I20260927 18:50:43.721302 15355 ts_tablet_manager.cc:1397] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:50:43.721417 15354 log.cc:824] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:43.723333 15354 tablet_bootstrap.cc:492] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb: No bootstrap required, opened a new log
I20260927 18:50:43.723560 15354 ts_tablet_manager.cc:1397] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:50:43.723944 15353 raft_consensus.cc:348] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a9b61365ff427686a96bd5c79da965" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42791 } } peers { permanent_uuid: "284a2fd1811d40f09307078c1e5d6cfb" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 44247 } } peers { permanent_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33139 } }
I20260927 18:50:43.724191 15353 raft_consensus.cc:374] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:43.724246 15353 raft_consensus.cc:729] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a9b61365ff427686a96bd5c79da965, State: Initialized, Role: FOLLOWER
I20260927 18:50:43.724694 15353 consensus_queue.cc:260] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a9b61365ff427686a96bd5c79da965" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42791 } } peers { permanent_uuid: "284a2fd1811d40f09307078c1e5d6cfb" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 44247 } } peers { permanent_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33139 } }
I20260927 18:50:43.725409 15336 heartbeater.cc:503] Master 127.29.159.126:44733 was elected leader, sending a full tablet report...
I20260927 18:50:43.725746 15353 ts_tablet_manager.cc:1428] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:50:43.726095 15355 raft_consensus.cc:348] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a9b61365ff427686a96bd5c79da965" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42791 } } peers { permanent_uuid: "284a2fd1811d40f09307078c1e5d6cfb" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 44247 } } peers { permanent_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33139 } }
I20260927 18:50:43.726334 15355 raft_consensus.cc:374] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:43.726393 15355 raft_consensus.cc:729] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bbd0f3b2fb64786b2b691e79ac3be18, State: Initialized, Role: FOLLOWER
I20260927 18:50:43.726830 15355 consensus_queue.cc:260] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a9b61365ff427686a96bd5c79da965" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42791 } } peers { permanent_uuid: "284a2fd1811d40f09307078c1e5d6cfb" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 44247 } } peers { permanent_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33139 } }
I20260927 18:50:43.727260 15354 raft_consensus.cc:348] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a9b61365ff427686a96bd5c79da965" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42791 } } peers { permanent_uuid: "284a2fd1811d40f09307078c1e5d6cfb" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 44247 } } peers { permanent_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33139 } }
I20260927 18:50:43.727550 15354 raft_consensus.cc:374] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:43.727553 15068 heartbeater.cc:503] Master 127.29.159.126:44733 was elected leader, sending a full tablet report...
I20260927 18:50:43.727602 15354 raft_consensus.cc:729] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284a2fd1811d40f09307078c1e5d6cfb, State: Initialized, Role: FOLLOWER
I20260927 18:50:43.727859 15355 ts_tablet_manager.cc:1428] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 18:50:43.728025 15354 consensus_queue.cc:260] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a9b61365ff427686a96bd5c79da965" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42791 } } peers { permanent_uuid: "284a2fd1811d40f09307078c1e5d6cfb" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 44247 } } peers { permanent_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33139 } }
I20260927 18:50:43.728680 15202 heartbeater.cc:503] Master 127.29.159.126:44733 was elected leader, sending a full tablet report...
I20260927 18:50:43.728971 15354 ts_tablet_manager.cc:1428] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:50:43.796543 15360 raft_consensus.cc:482] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:43.796756 15360 raft_consensus.cc:504] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a9b61365ff427686a96bd5c79da965" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42791 } } peers { permanent_uuid: "284a2fd1811d40f09307078c1e5d6cfb" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 44247 } } peers { permanent_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33139 } }
I20260927 18:50:43.797969 15360 leader_election.cc:288] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88a9b61365ff427686a96bd5c79da965 (127.29.159.67:42791), 284a2fd1811d40f09307078c1e5d6cfb (127.29.159.66:44247)
I20260927 18:50:43.803334 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.798249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:46269 (local address 127.29.159.67:42791)
0927 18:50:43.798397 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:43.798402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:43.798660 (+ 258us) server_negotiation.cc:407] Connection header received
0927 18:50:43.798829 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:43.798832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:43.798891 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:43.798978 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:43.799411 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.800174 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.802058 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.802688 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.802820 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.802873 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:43.803123 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 18:50:43.803205 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:50:43.803332 15362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.798170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:42791 (local address 127.29.159.65:46269)
0927 18:50:43.798556 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:43.798575 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:43.798699 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:43.799015 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:43.799018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:43.799060 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:43.799252 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.799258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.800292 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.800295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.801911 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.801924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.802845 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.802850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.802965 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.802994 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:50:43.803075 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:50:43.803153 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":169,"spinlock_wait_cycles":7040,"thread_start_us":121,"threads_started":1}
I20260927 18:50:43.803629 15352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.798374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:58163 (local address 127.29.159.66:44247)
0927 18:50:43.798504 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:43.798509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:43.798723 (+ 214us) server_negotiation.cc:407] Connection header received
0927 18:50:43.798839 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:43.798862 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:43.798911 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:43.798986 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:43.799363 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.800174 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.802058 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.802706 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.802828 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.802901 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:43.803274 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 18:50:43.803361 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:50:43.803786 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.798195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:44247 (local address 127.29.159.65:58163)
0927 18:50:43.798636 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:43.798648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:43.798725 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:43.799015 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:43.799018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:43.799059 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:43.799246 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.799254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.800292 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.800295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.801920 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:43.801925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.803001 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.803005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:43.803139 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.803162 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:50:43.803236 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:50:43.803305 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":156,"spinlock_wait_cycles":67584,"thread_start_us":151,"threads_started":1}
I20260927 18:50:43.804278 15290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706301ec91b48c1881e221bfec2521b" candidate_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a9b61365ff427686a96bd5c79da965" is_pre_election: true
I20260927 18:50:43.804500 15156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706301ec91b48c1881e221bfec2521b" candidate_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284a2fd1811d40f09307078c1e5d6cfb" is_pre_election: true
I20260927 18:50:43.804646 15290 raft_consensus.cc:2454] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bbd0f3b2fb64786b2b691e79ac3be18 in term 0.
I20260927 18:50:43.804744 15156 raft_consensus.cc:2454] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bbd0f3b2fb64786b2b691e79ac3be18 in term 0.
I20260927 18:50:43.805200 14955 leader_election.cc:302] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 284a2fd1811d40f09307078c1e5d6cfb, 7bbd0f3b2fb64786b2b691e79ac3be18, 88a9b61365ff427686a96bd5c79da965; no voters:
I20260927 18:50:43.805527 15360 raft_consensus.cc:2781] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:50:43.805626 15360 raft_consensus.cc:482] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:50:43.805688 15360 raft_consensus.cc:3037] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:43.806731 15360 raft_consensus.cc:504] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a9b61365ff427686a96bd5c79da965" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42791 } } peers { permanent_uuid: "284a2fd1811d40f09307078c1e5d6cfb" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 44247 } } peers { permanent_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33139 } }
I20260927 18:50:43.807088 15360 leader_election.cc:288] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [CANDIDATE]: Term 1 election: Requested vote from peers 88a9b61365ff427686a96bd5c79da965 (127.29.159.67:42791), 284a2fd1811d40f09307078c1e5d6cfb (127.29.159.66:44247)
I20260927 18:50:43.807466 15156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706301ec91b48c1881e221bfec2521b" candidate_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284a2fd1811d40f09307078c1e5d6cfb"
I20260927 18:50:43.807580 15290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706301ec91b48c1881e221bfec2521b" candidate_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a9b61365ff427686a96bd5c79da965"
I20260927 18:50:43.807641 15156 raft_consensus.cc:3037] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:43.807716 15290 raft_consensus.cc:3037] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:43.808681 15156 raft_consensus.cc:2454] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bbd0f3b2fb64786b2b691e79ac3be18 in term 1.
I20260927 18:50:43.808714 15290 raft_consensus.cc:2454] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bbd0f3b2fb64786b2b691e79ac3be18 in term 1.
I20260927 18:50:43.809017 14956 leader_election.cc:302] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284a2fd1811d40f09307078c1e5d6cfb, 7bbd0f3b2fb64786b2b691e79ac3be18; no voters:
I20260927 18:50:43.809201 15360 raft_consensus.cc:2781] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:43.809412 15360 raft_consensus.cc:686] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [term 1 LEADER]: Becoming Leader. State: Replica: 7bbd0f3b2fb64786b2b691e79ac3be18, State: Running, Role: LEADER
I20260927 18:50:43.809753 15360 consensus_queue.cc:237] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a9b61365ff427686a96bd5c79da965" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42791 } } peers { permanent_uuid: "284a2fd1811d40f09307078c1e5d6cfb" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 44247 } } peers { permanent_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33139 } }
I20260927 18:50:43.812990 14877 catalog_manager.cc:5495] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bbd0f3b2fb64786b2b691e79ac3be18 (127.29.159.65). New cstate: current_term: 1 leader_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a9b61365ff427686a96bd5c79da965" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "284a2fd1811d40f09307078c1e5d6cfb" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 44247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bbd0f3b2fb64786b2b691e79ac3be18" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 33139 } health_report { overall_health: HEALTHY } } }
I20260927 18:50:43.824564 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.820919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49952 (local address 127.29.159.65:33139)
0927 18:50:43.821049 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:43.821055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:43.821091 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:50:43.821274 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:43.821277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:43.821340 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:43.821449 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:43.821703 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.822210 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.823065 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:43.823294 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:43.823404 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:43.823739 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:43.824150 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:43.824285 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:43.824325 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 18:50:43.824381 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:50:43.830350 15156 raft_consensus.cc:1261] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb [term 1 FOLLOWER]: Refusing update from remote peer 7bbd0f3b2fb64786b2b691e79ac3be18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:43.830390 15290 raft_consensus.cc:1261] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965 [term 1 FOLLOWER]: Refusing update from remote peer 7bbd0f3b2fb64786b2b691e79ac3be18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:43.831074 15364 consensus_queue.cc:1035] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a9b61365ff427686a96bd5c79da965" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:43.831308 15360 consensus_queue.cc:1035] T 8706301ec91b48c1881e221bfec2521b P 7bbd0f3b2fb64786b2b691e79ac3be18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "284a2fd1811d40f09307078c1e5d6cfb" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:43.838924 15368 mvcc.cc:205] Tried to move back new op lower bound from 7334031539522187264 to 7334031539447418880. Current Snapshot: MvccSnapshot[applied={T|T < 7334031539522187264}]
I20260927 18:50:43.848536 14876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:43.848724 14876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:43.851270 14876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc14a945fc824af199b4ece821bc9a75: Sending DeleteTablet for 3 replicas of tablet 8706301ec91b48c1881e221bfec2521b
I20260927 18:50:43.852118 15270 tablet_service.cc:1514] Processing DeleteTablet for tablet 8706301ec91b48c1881e221bfec2521b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:43 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:34642
I20260927 18:50:43.852442 15136 tablet_service.cc:1514] Processing DeleteTablet for tablet 8706301ec91b48c1881e221bfec2521b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:43 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:34656
I20260927 18:50:43.852662 15002 tablet_service.cc:1514] Processing DeleteTablet for tablet 8706301ec91b48c1881e221bfec2521b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:43 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:49938
I20260927 18:50:43.852752 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 14939
I20260927 18:50:43.853533 15377 tablet_replica.cc:330] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965: stopping tablet replica
I20260927 18:50:43.853881 15377 raft_consensus.cc:2229] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:43.854230 15377 raft_consensus.cc:2258] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:43.856094 15377 ts_tablet_manager.cc:1905] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:43.857662 15378 tablet_replica.cc:330] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb: stopping tablet replica
I20260927 18:50:43.857970 15378 raft_consensus.cc:2229] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:43.858279 15378 raft_consensus.cc:2258] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:43.859658 15378 ts_tablet_manager.cc:1905] T 8706301ec91b48c1881e221bfec2521b P 284a2fd1811d40f09307078c1e5d6cfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:43.860081 15377 ts_tablet_manager.cc:1918] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:43.860183 15377 log.cc:1196] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/8706301ec91b48c1881e221bfec2521b
I20260927 18:50:43.860508 15377 ts_tablet_manager.cc:1939] T 8706301ec91b48c1881e221bfec2521b P 88a9b61365ff427686a96bd5c79da965: Deleting consensus metadata
W20260927 18:50:43.860807 14862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:33139 (error 108)
I20260927 18:50:43.861014 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 15073
I20260927 18:50:43.861869 14861 catalog_manager.cc:4841] TS 88a9b61365ff427686a96bd5c79da965 (127.29.159.67:42791): tablet 8706301ec91b48c1881e221bfec2521b (table test-table [id=c6f03ff1bddf48f48f4f319c889effe8]) successfully deleted
I20260927 18:50:43.863462 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.863076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:33139 (local address 127.0.0.1:49968)
0927 18:50:43.863237 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:43.863321 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:33139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 18:50:43.863636 14862 catalog_manager.cc:4584] TS 7bbd0f3b2fb64786b2b691e79ac3be18 (127.29.159.65:33139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8706301ec91b48c1881e221bfec2521b: Network error: Client connection negotiation failed: client connection to 127.29.159.65:33139: connect: Connection refused (error 111)
I20260927 18:50:43.867220 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 15207
I20260927 18:50:43.868028 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:43.867322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:44247 (local address 127.0.0.1:34662)
0927 18:50:43.867853 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:43.867928 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.66:44247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438}
I20260927 18:50:43.872934 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 14845
2026-09-27T18:50:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2339 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T18:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:43Z Disabled control of system clock
I20260927 18:50:43.903774 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44683
--webserver_interface=127.29.159.126
--webserver_port=0
--builtin_ntp_servers=127.29.159.84:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:44683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:50:44.015105 15383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:44.015383 15383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:44.015473 15383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:44.019646 15383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:44.019706 15383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:44.019722 15383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:44.019739 15383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:44.019755 15383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:44.024665 15383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:44683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:44683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:44.025734 15383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:44.027082 15383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:44.035710 15388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:44.035902 15383 server_base.cc:1034] running on GCE node
W20260927 18:50:44.035727 15391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:44.035759 15389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:44.036671 15383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:44.037897 15383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:44.039098 15383 hybrid_clock.cc:648] HybridClock initialized: now 1790535044039091 us; error 32 us; skew 500 ppm
I20260927 18:50:44.041136 15383 webserver.cc:460] Webserver started at http://127.29.159.126:37039/ using document root <none> and password file <none>
I20260927 18:50:44.041819 15383 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:44.041929 15383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:44.042186 15383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:44.044461 15383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "890d8903e7cd40cf9d13d118ec6a31fb"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "a3dd161d7240583dd412fed461c44819"
server_key_iv: "64c1e9da5cd396e52b1ef6c370cf4ff2"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.045166 15383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "890d8903e7cd40cf9d13d118ec6a31fb"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "a3dd161d7240583dd412fed461c44819"
server_key_iv: "64c1e9da5cd396e52b1ef6c370cf4ff2"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.049250 15383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 18:50:44.051995 15397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:44.053303 15383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:44.053471 15383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "890d8903e7cd40cf9d13d118ec6a31fb"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "a3dd161d7240583dd412fed461c44819"
server_key_iv: "64c1e9da5cd396e52b1ef6c370cf4ff2"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.053603 15383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:44.073529 15383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:44.074174 15383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:44.074326 15383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:44.081394 15383 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:44683
I20260927 18:50:44.081398 15449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:44683 every 8 connection(s)
I20260927 18:50:44.082372 15383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:44.085505 15450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:44.089919 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 15383
I20260927 18:50:44.090049 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:50:44.090335 30333 external_mini_cluster.cc:1428] Setting key 89f73c37586a7217fe38d4fe4bee6233
I20260927 18:50:44.091058 15450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb: Bootstrap starting.
I20260927 18:50:44.093371 15450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:44.094113 15450 log.cc:824] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:44.096101 15450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb: No bootstrap required, opened a new log
I20260927 18:50:44.099836 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.091763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.29.159.126:44683)
0927 18:50:44.092398 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:44.092415 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:44.092450 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:50:44.093058 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:44.093074 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:44.093507 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:44.093838 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:44.094357 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.095560 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.096504 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.097031 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.097208 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.098072 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:44.098077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:44.098090 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:44.098497 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:44.098550 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:44.098569 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:44.098680 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:44.098822 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 18:50:44.099075 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":124,"threads_started":1}
I20260927 18:50:44.099813 15450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d8903e7cd40cf9d13d118ec6a31fb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44683 } }
I20260927 18:50:44.100085 15450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:44.100136 15450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890d8903e7cd40cf9d13d118ec6a31fb, State: Initialized, Role: FOLLOWER
I20260927 18:50:44.100569 15450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d8903e7cd40cf9d13d118ec6a31fb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44683 } }
I20260927 18:50:44.100704 15450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:44.100760 15450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:44.100845 15450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:44.101686 15450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d8903e7cd40cf9d13d118ec6a31fb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44683 } }
I20260927 18:50:44.102075 15450 leader_election.cc:302] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 890d8903e7cd40cf9d13d118ec6a31fb; no voters:
I20260927 18:50:44.102417 15450 leader_election.cc:288] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:44.102612 15455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:44.102867 15455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [term 1 LEADER]: Becoming Leader. State: Replica: 890d8903e7cd40cf9d13d118ec6a31fb, State: Running, Role: LEADER
I20260927 18:50:44.103168 15455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d8903e7cd40cf9d13d118ec6a31fb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44683 } }
I20260927 18:50:44.103734 15450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:44.104799 15456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "890d8903e7cd40cf9d13d118ec6a31fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d8903e7cd40cf9d13d118ec6a31fb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44683 } } }
I20260927 18:50:44.104946 15456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:44.105222 15457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 890d8903e7cd40cf9d13d118ec6a31fb. Latest consensus state: current_term: 1 leader_uuid: "890d8903e7cd40cf9d13d118ec6a31fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d8903e7cd40cf9d13d118ec6a31fb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 44683 } } }
I20260927 18:50:44.105321 15457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:44.105656 15464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:44.108980 15464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:44.114084 15464 catalog_manager.cc:1347] Generated new cluster ID: 537abc87e1e64577b7ffad525a9c154f
I20260927 18:50:44.114149 15464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:44.124740 15464 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:44.125989 15464 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:44.140611 15464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb: Generated new TSK 0
I20260927 18:50:44.141467 15464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:44.145531 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44683
--builtin_ntp_servers=127.29.159.84:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:50:44.264303 15474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:44.264587 15474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:44.264654 15474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:44.268757 15474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:44.268816 15474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:44.268905 15474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:50:44.273850 15474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:44.275009 15474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:44.276481 15474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:44.286058 15482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:44.286103 15474 server_base.cc:1034] running on GCE node
W20260927 18:50:44.286120 15479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:44.286159 15480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:44.286657 15474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:44.287339 15474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:44.288528 15474 hybrid_clock.cc:648] HybridClock initialized: now 1790535044288494 us; error 49 us; skew 500 ppm
I20260927 18:50:44.290367 15474 webserver.cc:460] Webserver started at http://127.29.159.65:37547/ using document root <none> and password file <none>
I20260927 18:50:44.290946 15474 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:44.291044 15474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:44.291270 15474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:44.293342 15474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "f190e88725a442f4a37a5fd9a47a4508"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "796baa606dc82a29d55fce598092a820"
server_key_iv: "7ef29f129c9f93d5ea9959b018abfcc1"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.293948 15474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "f190e88725a442f4a37a5fd9a47a4508"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "796baa606dc82a29d55fce598092a820"
server_key_iv: "7ef29f129c9f93d5ea9959b018abfcc1"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.298173 15474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:50:44.300616 15488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:44.301815 15474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:50:44.301939 15474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "f190e88725a442f4a37a5fd9a47a4508"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "796baa606dc82a29d55fce598092a820"
server_key_iv: "7ef29f129c9f93d5ea9959b018abfcc1"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.302062 15474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:44.315478 15474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:44.316257 15474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:44.316480 15474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:44.317147 15474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:44.318157 15474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:44.318238 15474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:44.318342 15474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:44.318397 15474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:44.328341 15474 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:35939
I20260927 18:50:44.328358 15601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:35939 every 8 connection(s)
I20260927 18:50:44.329324 15474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:50:44.332325 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 15474
I20260927 18:50:44.332453 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:50:44.332768 30333 external_mini_cluster.cc:1428] Setting key 5341804a47e20003ff75e473aab8820a
I20260927 18:50:44.334702 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44683
--builtin_ntp_servers=127.29.159.84:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:50:44.338923 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.331292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:36083 (local address 127.29.159.126:44683)
0927 18:50:44.331612 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:44.331616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:44.332676 (+ 1060us) server_negotiation.cc:407] Connection header received
0927 18:50:44.334124 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:44.334128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:44.334182 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:44.334271 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:44.335265 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.335822 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.336980 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.337212 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.337326 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.338432 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:44.338435 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:44.338438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:44.338457 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:44.338480 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:44.338489 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:44.338569 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:44.338722 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 18:50:44.338783 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260927 18:50:44.339855 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.331742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44683 (local address 127.29.159.65:36083)
0927 18:50:44.332487 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:44.332551 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:44.333907 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:44.334464 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:44.334472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:44.334558 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:44.335127 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.335140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.335956 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.335959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.336843 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.336851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.337365 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.337369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.337539 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.338162 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:50:44.338163 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:44.338203 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:50:44.338266 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:44.338571 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:44.338583 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:44.338591 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:50:44.338722 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 18:50:44.339011 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":103,"threads_started":1}
I20260927 18:50:44.341130 15602 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44683
I20260927 18:50:44.341373 15602 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:44.341921 15602 heartbeater.cc:511] Master 127.29.159.126:44683 requested a full tablet report, sending...
I20260927 18:50:44.343634 15414 ts_manager.cc:194] Registered new tserver with Master: f190e88725a442f4a37a5fd9a47a4508 (127.29.159.65:35939)
I20260927 18:50:44.345355 15414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.65:36083
W20260927 18:50:44.445801 15605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:44.446041 15605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:44.446079 15605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:44.449958 15605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:44.450001 15605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:44.450050 15605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:50:44.454614 15605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:44.455605 15605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:44.456882 15605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:44.466411 15611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:44.466425 15613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:44.466413 15610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:44.466485 15605 server_base.cc:1034] running on GCE node
I20260927 18:50:44.467093 15605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:44.467736 15605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:44.468895 15605 hybrid_clock.cc:648] HybridClock initialized: now 1790535044468869 us; error 38 us; skew 500 ppm
I20260927 18:50:44.470660 15605 webserver.cc:460] Webserver started at http://127.29.159.66:37847/ using document root <none> and password file <none>
I20260927 18:50:44.471194 15605 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:44.471252 15605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:44.471467 15605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:44.473482 15605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "324319386da6425eafce632cdd225399"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "beca7505cdd3f9284678f1e78bb376c0"
server_key_iv: "b86a9b8d6c946706308c1c64dbbb5b03"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.474016 15605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "324319386da6425eafce632cdd225399"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "beca7505cdd3f9284678f1e78bb376c0"
server_key_iv: "b86a9b8d6c946706308c1c64dbbb5b03"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.477573 15605 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:50:44.479908 15619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:44.481192 15605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:44.481297 15605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "324319386da6425eafce632cdd225399"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "beca7505cdd3f9284678f1e78bb376c0"
server_key_iv: "b86a9b8d6c946706308c1c64dbbb5b03"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.481381 15605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:44.502619 15605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:44.503230 15605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:44.503379 15605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:44.503980 15605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:44.504983 15605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:44.505048 15605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:44.505113 15605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:44.505147 15605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:44.513945 15605 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:35481
I20260927 18:50:44.513969 15732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:35481 every 8 connection(s)
I20260927 18:50:44.514822 15605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:50:44.520910 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 15605
I20260927 18:50:44.521040 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:50:44.521343 30333 external_mini_cluster.cc:1428] Setting key 94e05f2fe7f9d3026c52dbcda1995cea
I20260927 18:50:44.523279 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44683
--builtin_ntp_servers=127.29.159.84:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:50:44.524374 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.516665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:51633 (local address 127.29.159.126:44683)
0927 18:50:44.516822 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:44.516826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:44.517832 (+ 1006us) server_negotiation.cc:407] Connection header received
0927 18:50:44.519139 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:44.519142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:44.519188 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:44.519260 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:44.520193 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.520722 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.522277 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.522595 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.522713 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.523853 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:44.523857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:44.523859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:44.523872 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:44.523896 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:44.523904 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:44.524005 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:44.524164 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 18:50:44.524238 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:50:44.525162 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.517036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44683 (local address 127.29.159.66:51633)
0927 18:50:44.517686 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:44.517726 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:44.518958 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:44.519416 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:44.519470 (+ 54us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:44.519571 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:44.520072 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.520085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.520843 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.520847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.522163 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.522171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.522747 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.522751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.522911 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.523606 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:50:44.523607 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:44.523645 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:50:44.523705 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:44.524009 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:44.524020 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:44.524028 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:50:44.524164 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 18:50:44.524429 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":99,"threads_started":1}
I20260927 18:50:44.526360 15733 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44683
I20260927 18:50:44.526595 15733 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:44.527091 15733 heartbeater.cc:511] Master 127.29.159.126:44683 requested a full tablet report, sending...
I20260927 18:50:44.528162 15414 ts_manager.cc:194] Registered new tserver with Master: 324319386da6425eafce632cdd225399 (127.29.159.66:35481)
I20260927 18:50:44.528753 15414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.66:51633
W20260927 18:50:44.633936 15736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:44.634207 15736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:44.634276 15736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:44.638365 15736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:44.638454 15736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:44.638545 15736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:50:44.643859 15736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:44683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:44.644918 15736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:44.646287 15736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:44.655859 15742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:44.655920 15744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:44.655988 15736 server_base.cc:1034] running on GCE node
W20260927 18:50:44.655859 15741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:44.656520 15736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:44.657215 15736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:44.658397 15736 hybrid_clock.cc:648] HybridClock initialized: now 1790535044658379 us; error 42 us; skew 500 ppm
I20260927 18:50:44.660162 15736 webserver.cc:460] Webserver started at http://127.29.159.67:39469/ using document root <none> and password file <none>
I20260927 18:50:44.660743 15736 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:44.660836 15736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:44.661082 15736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:44.663089 15736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "f083633ee17446dbacaab5a5a98f640c"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "9a4a6ecb87c41505d412d52c6d51e478"
server_key_iv: "c710816b33237168aeb0e5f8bf609b5e"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.663679 15736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "f083633ee17446dbacaab5a5a98f640c"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "9a4a6ecb87c41505d412d52c6d51e478"
server_key_iv: "c710816b33237168aeb0e5f8bf609b5e"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.667151 15736 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:50:44.669528 15750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:44.670701 15736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 18:50:44.670857 15736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "f083633ee17446dbacaab5a5a98f640c"
format_stamp: "Formatted at 2026-09-27 18:50:44 on dist-test-slave-8tg2"
server_key: "9a4a6ecb87c41505d412d52c6d51e478"
server_key_iv: "c710816b33237168aeb0e5f8bf609b5e"
server_key_version: "encryptionkey@0"
I20260927 18:50:44.670974 15736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:44.686810 15736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:44.687525 15736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:44.687709 15736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:44.688297 15736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:44.689221 15736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:44.689291 15736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:44.689363 15736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:44.689415 15736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:44.699170 15736 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:41861
I20260927 18:50:44.699194 15863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:41861 every 8 connection(s)
I20260927 18:50:44.700137 15736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:50:44.709911 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 15736
I20260927 18:50:44.710012 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.702013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:46985 (local address 127.29.159.126:44683)
0927 18:50:44.702154 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:44.702158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:44.703199 (+ 1041us) server_negotiation.cc:407] Connection header received
0927 18:50:44.704861 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:44.704864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:44.704924 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:44.705033 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:44.706156 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.706812 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.708071 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.708304 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.708418 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.709480 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:44.709483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:44.709486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:44.709499 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:44.709529 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:44.709537 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:44.709617 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:44.709786 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 18:50:44.709849 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:50:44.710036 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:50:44.710394 30333 external_mini_cluster.cc:1428] Setting key b06044e1adee3f2ffe38ff06477bce52
I20260927 18:50:44.710922 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.702423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:44683 (local address 127.29.159.67:46985)
0927 18:50:44.703051 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:44.703090 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:44.704638 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:44.705222 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:44.705230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:44.705319 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:44.705889 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.705902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.706946 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.706949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.707947 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.707956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.708460 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.708464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.708636 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.709219 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:50:44.709220 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:44.709259 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:50:44.709324 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:44.709618 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:44.709629 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:44.709637 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:50:44.709776 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 18:50:44.710057 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260927 18:50:44.712064 15864 heartbeater.cc:347] Connected to a master server at 127.29.159.126:44683
I20260927 18:50:44.712299 15864 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:44.712878 15864 heartbeater.cc:511] Master 127.29.159.126:44683 requested a full tablet report, sending...
I20260927 18:50:44.713820 15414 ts_manager.cc:194] Registered new tserver with Master: f083633ee17446dbacaab5a5a98f640c (127.29.159.67:41861)
I20260927 18:50:44.714394 15414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.67:46985
I20260927 18:50:44.724875 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:50:44.730724 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.726677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.29.159.126:44683)
0927 18:50:44.726823 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:44.726827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:44.727015 (+ 188us) server_negotiation.cc:407] Connection header received
0927 18:50:44.727161 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:44.727164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:44.727207 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:44.727297 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:44.727721 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.728257 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.729124 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.729360 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.729481 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.730229 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:44.730233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:44.730235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:44.730247 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:44.730267 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:44.730282 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:44.730353 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:44.730533 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 18:50:44.730585 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:50:44.733330 15414 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:50:44.736315 15414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:50:44.750228 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.744422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38488 (local address 127.29.159.67:41861)
0927 18:50:44.745219 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:44.745233 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:44.745258 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:50:44.745335 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:44.745340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:44.745551 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:44.745711 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:44.746018 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.747147 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.748882 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.749537 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.749663 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.749867 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:44.749951 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 18:50:44.750046 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":94,"threads_started":1}
I20260927 18:50:44.750240 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.744324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:35939 (local address 127.0.0.1:54866)
0927 18:50:44.745288 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:44.745304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:44.745397 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:44.746374 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:44.746380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:44.746402 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:44.746545 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.746552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.747855 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.747858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.748810 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.748818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.749751 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.749755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.749846 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.749872 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:50:44.749922 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 18:50:44.750054 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":85,"thread_start_us":82,"threads_started":1}
I20260927 18:50:44.750254 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.744290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:41861 (local address 127.0.0.1:38488)
0927 18:50:44.744687 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:44.744741 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:44.744880 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:44.745711 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:44.745717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:44.745739 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:44.745884 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.745892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.747270 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.747273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.748735 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.748745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.749661 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.749664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.749772 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.749806 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:50:44.749921 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:50:44.750041 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":52,"thread_start_us":87,"threads_started":1}
I20260927 18:50:44.750926 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.745023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:35481 (local address 127.0.0.1:38468)
0927 18:50:44.745565 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:44.745578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:44.745652 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:44.746440 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:44.746442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:44.746453 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:44.746556 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.746561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.747905 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.747909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.749102 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.749111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.750118 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.750121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.750208 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.750228 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:50:44.750276 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 18:50:44.750569 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":61,"thread_start_us":84,"threads_started":1}
I20260927 18:50:44.751135 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.745223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.29.159.65:35939)
0927 18:50:44.745896 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:44.745915 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:44.745935 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:50:44.746035 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:44.746040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:44.746226 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:44.746373 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:44.746661 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.747733 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.748926 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.749613 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.750606 (+ 993us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.750788 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:44.750841 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 18:50:44.750934 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":106,"threads_started":1}
I20260927 18:50:44.752390 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.745612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.29.159.66:35481)
0927 18:50:44.746023 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:44.746042 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:44.746062 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:50:44.746122 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:44.746125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:44.746273 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:44.746696 (+ 423us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:44.746816 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.747774 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.749224 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.749990 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.751166 (+ 1176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.752114 (+ 948us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:44.752162 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 18:50:44.752244 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":111,"threads_started":1}
I20260927 18:50:44.753619 15798 tablet_service.cc:1467] Processing CreateTablet for tablet 98b13095eb4b4bd0a9a1c284dcc6302c (DEFAULT_TABLE table=test-table [id=f13e4ae8bdf547a892440595747d4b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:44.753801 15536 tablet_service.cc:1467] Processing CreateTablet for tablet 98b13095eb4b4bd0a9a1c284dcc6302c (DEFAULT_TABLE table=test-table [id=f13e4ae8bdf547a892440595747d4b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:44.754698 15798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b13095eb4b4bd0a9a1c284dcc6302c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:44.754810 15536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b13095eb4b4bd0a9a1c284dcc6302c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:44.754923 15667 tablet_service.cc:1467] Processing CreateTablet for tablet 98b13095eb4b4bd0a9a1c284dcc6302c (DEFAULT_TABLE table=test-table [id=f13e4ae8bdf547a892440595747d4b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:44.755857 15667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b13095eb4b4bd0a9a1c284dcc6302c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:44.761184 15881 tablet_bootstrap.cc:492] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c: Bootstrap starting.
I20260927 18:50:44.762627 15882 tablet_bootstrap.cc:492] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508: Bootstrap starting.
I20260927 18:50:44.763350 15881 tablet_bootstrap.cc:654] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:44.763852 15883 tablet_bootstrap.cc:492] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399: Bootstrap starting.
I20260927 18:50:44.764195 15881 log.cc:824] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:44.764650 15882 tablet_bootstrap.cc:654] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:44.765434 15882 log.cc:824] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:44.765913 15883 tablet_bootstrap.cc:654] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:44.766230 15881 tablet_bootstrap.cc:492] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c: No bootstrap required, opened a new log
I20260927 18:50:44.766430 15881 ts_tablet_manager.cc:1397] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:50:44.766702 15883 log.cc:824] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:44.767314 15882 tablet_bootstrap.cc:492] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508: No bootstrap required, opened a new log
I20260927 18:50:44.767565 15882 ts_tablet_manager.cc:1397] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 18:50:44.768676 15883 tablet_bootstrap.cc:492] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399: No bootstrap required, opened a new log
I20260927 18:50:44.768854 15883 ts_tablet_manager.cc:1397] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:50:44.769894 15881 raft_consensus.cc:348] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f083633ee17446dbacaab5a5a98f640c" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41861 } } peers { permanent_uuid: "f190e88725a442f4a37a5fd9a47a4508" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 35939 } } peers { permanent_uuid: "324319386da6425eafce632cdd225399" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35481 } }
I20260927 18:50:44.770133 15881 raft_consensus.cc:374] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:44.770185 15881 raft_consensus.cc:729] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f083633ee17446dbacaab5a5a98f640c, State: Initialized, Role: FOLLOWER
I20260927 18:50:44.770635 15881 consensus_queue.cc:260] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f083633ee17446dbacaab5a5a98f640c" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41861 } } peers { permanent_uuid: "f190e88725a442f4a37a5fd9a47a4508" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 35939 } } peers { permanent_uuid: "324319386da6425eafce632cdd225399" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35481 } }
I20260927 18:50:44.771255 15864 heartbeater.cc:503] Master 127.29.159.126:44683 was elected leader, sending a full tablet report...
I20260927 18:50:44.771633 15881 ts_tablet_manager.cc:1428] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:50:44.771829 15882 raft_consensus.cc:348] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f083633ee17446dbacaab5a5a98f640c" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41861 } } peers { permanent_uuid: "f190e88725a442f4a37a5fd9a47a4508" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 35939 } } peers { permanent_uuid: "324319386da6425eafce632cdd225399" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35481 } }
I20260927 18:50:44.772056 15882 raft_consensus.cc:374] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:44.772100 15882 raft_consensus.cc:729] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f190e88725a442f4a37a5fd9a47a4508, State: Initialized, Role: FOLLOWER
I20260927 18:50:44.772544 15882 consensus_queue.cc:260] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f083633ee17446dbacaab5a5a98f640c" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41861 } } peers { permanent_uuid: "f190e88725a442f4a37a5fd9a47a4508" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 35939 } } peers { permanent_uuid: "324319386da6425eafce632cdd225399" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35481 } }
I20260927 18:50:44.772948 15883 raft_consensus.cc:348] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f083633ee17446dbacaab5a5a98f640c" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41861 } } peers { permanent_uuid: "f190e88725a442f4a37a5fd9a47a4508" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 35939 } } peers { permanent_uuid: "324319386da6425eafce632cdd225399" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35481 } }
I20260927 18:50:44.773175 15883 raft_consensus.cc:374] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:44.773169 15602 heartbeater.cc:503] Master 127.29.159.126:44683 was elected leader, sending a full tablet report...
I20260927 18:50:44.773221 15883 raft_consensus.cc:729] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324319386da6425eafce632cdd225399, State: Initialized, Role: FOLLOWER
I20260927 18:50:44.773478 15882 ts_tablet_manager.cc:1428] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:50:44.773624 15883 consensus_queue.cc:260] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f083633ee17446dbacaab5a5a98f640c" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41861 } } peers { permanent_uuid: "f190e88725a442f4a37a5fd9a47a4508" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 35939 } } peers { permanent_uuid: "324319386da6425eafce632cdd225399" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35481 } }
I20260927 18:50:44.774586 15883 ts_tablet_manager.cc:1428] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260927 18:50:44.774932 15733 heartbeater.cc:503] Master 127.29.159.126:44683 was elected leader, sending a full tablet report...
I20260927 18:50:44.819527 15887 raft_consensus.cc:482] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:44.819753 15887 raft_consensus.cc:504] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f083633ee17446dbacaab5a5a98f640c" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41861 } } peers { permanent_uuid: "f190e88725a442f4a37a5fd9a47a4508" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 35939 } } peers { permanent_uuid: "324319386da6425eafce632cdd225399" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35481 } }
I20260927 18:50:44.821022 15887 leader_election.cc:288] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f190e88725a442f4a37a5fd9a47a4508 (127.29.159.65:35939), 324319386da6425eafce632cdd225399 (127.29.159.66:35481)
I20260927 18:50:44.825119 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.821514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:55269 (local address 127.29.159.65:35939)
0927 18:50:44.821633 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:44.821637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:44.821651 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:50:44.821704 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:44.821707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:44.821756 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:44.821829 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:44.822264 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.822900 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.824090 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.824510 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.824631 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.824686 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:44.824911 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 18:50:44.824988 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:50:44.825133 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.821181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:35939 (local address 127.29.159.67:55269)
0927 18:50:44.821406 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:44.821424 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:44.821524 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:44.821981 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:44.821984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:44.822002 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:44.822131 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.822139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.823021 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.823025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.823964 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.823978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.824633 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.824636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.824733 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.824762 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:50:44.824879 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 18:50:44.825006 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":63}
I20260927 18:50:44.825117 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.821371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:35045 (local address 127.29.159.66:35481)
0927 18:50:44.821500 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:44.821504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:44.821801 (+ 297us) server_negotiation.cc:407] Connection header received
0927 18:50:44.821997 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:44.822001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:44.822048 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:44.822127 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:44.822546 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.823063 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.824157 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.824537 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.824672 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.824722 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:44.824908 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 18:50:44.824989 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:50:44.825201 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.821284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:35481 (local address 127.29.159.67:35045)
0927 18:50:44.821715 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:44.821731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:44.821875 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:44.822180 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:44.822212 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:44.822237 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:44.822415 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.822423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.823172 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.823174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.824007 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:44.824017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.824681 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.824685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:44.824775 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.824791 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 18:50:44.824878 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:50:44.824988 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":4,"thread_start_us":118,"threads_started":1}
I20260927 18:50:44.825817 15556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b13095eb4b4bd0a9a1c284dcc6302c" candidate_uuid: "f083633ee17446dbacaab5a5a98f640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f190e88725a442f4a37a5fd9a47a4508" is_pre_election: true
I20260927 18:50:44.826012 15687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b13095eb4b4bd0a9a1c284dcc6302c" candidate_uuid: "f083633ee17446dbacaab5a5a98f640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324319386da6425eafce632cdd225399" is_pre_election: true
I20260927 18:50:44.826149 15556 raft_consensus.cc:2454] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f083633ee17446dbacaab5a5a98f640c in term 0.
I20260927 18:50:44.826261 15687 raft_consensus.cc:2454] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f083633ee17446dbacaab5a5a98f640c in term 0.
I20260927 18:50:44.826738 15752 leader_election.cc:302] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 324319386da6425eafce632cdd225399, f083633ee17446dbacaab5a5a98f640c, f190e88725a442f4a37a5fd9a47a4508; no voters:
I20260927 18:50:44.827013 15887 raft_consensus.cc:2781] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:50:44.827095 15887 raft_consensus.cc:482] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:50:44.827138 15887 raft_consensus.cc:3037] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:44.828239 15887 raft_consensus.cc:504] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f083633ee17446dbacaab5a5a98f640c" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41861 } } peers { permanent_uuid: "f190e88725a442f4a37a5fd9a47a4508" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 35939 } } peers { permanent_uuid: "324319386da6425eafce632cdd225399" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35481 } }
I20260927 18:50:44.828552 15887 leader_election.cc:288] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [CANDIDATE]: Term 1 election: Requested vote from peers f190e88725a442f4a37a5fd9a47a4508 (127.29.159.65:35939), 324319386da6425eafce632cdd225399 (127.29.159.66:35481)
I20260927 18:50:44.828877 15556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b13095eb4b4bd0a9a1c284dcc6302c" candidate_uuid: "f083633ee17446dbacaab5a5a98f640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f190e88725a442f4a37a5fd9a47a4508"
I20260927 18:50:44.828900 15687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b13095eb4b4bd0a9a1c284dcc6302c" candidate_uuid: "f083633ee17446dbacaab5a5a98f640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324319386da6425eafce632cdd225399"
I20260927 18:50:44.828985 15556 raft_consensus.cc:3037] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:44.829066 15687 raft_consensus.cc:3037] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:44.830034 15556 raft_consensus.cc:2454] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f083633ee17446dbacaab5a5a98f640c in term 1.
I20260927 18:50:44.830150 15687 raft_consensus.cc:2454] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f083633ee17446dbacaab5a5a98f640c in term 1.
I20260927 18:50:44.830361 15751 leader_election.cc:302] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f083633ee17446dbacaab5a5a98f640c, f190e88725a442f4a37a5fd9a47a4508; no voters:
W20260927 18:50:44.830433 15603 tablet.cc:2369] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:50:44.830526 15887 raft_consensus.cc:2781] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:44.830845 15887 raft_consensus.cc:686] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 1 LEADER]: Becoming Leader. State: Replica: f083633ee17446dbacaab5a5a98f640c, State: Running, Role: LEADER
I20260927 18:50:44.831178 15887 consensus_queue.cc:237] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f083633ee17446dbacaab5a5a98f640c" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41861 } } peers { permanent_uuid: "f190e88725a442f4a37a5fd9a47a4508" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 35939 } } peers { permanent_uuid: "324319386da6425eafce632cdd225399" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35481 } }
I20260927 18:50:44.834640 15414 catalog_manager.cc:5495] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c reported cstate change: term changed from 0 to 1, leader changed from <none> to f083633ee17446dbacaab5a5a98f640c (127.29.159.67). New cstate: current_term: 1 leader_uuid: "f083633ee17446dbacaab5a5a98f640c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f083633ee17446dbacaab5a5a98f640c" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f190e88725a442f4a37a5fd9a47a4508" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 35939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "324319386da6425eafce632cdd225399" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35481 } health_report { overall_health: UNKNOWN } } }
I20260927 18:50:44.848657 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.845398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.29.159.67:41861)
0927 18:50:44.845525 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:44.845529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:44.845605 (+ 76us) server_negotiation.cc:407] Connection header received
0927 18:50:44.845731 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:44.845734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:44.845784 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:44.845888 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:44.846078 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.846567 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.847368 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:44.847621 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:44.847698 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:44.847952 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:44.848345 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:44.848432 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:44.848492 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 18:50:44.848533 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:50:44.854372 15687 raft_consensus.cc:1261] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399 [term 1 FOLLOWER]: Refusing update from remote peer f083633ee17446dbacaab5a5a98f640c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:44.854372 15556 raft_consensus.cc:1261] T 98b13095eb4b4bd0a9a1c284dcc6302c P f190e88725a442f4a37a5fd9a47a4508 [term 1 FOLLOWER]: Refusing update from remote peer f083633ee17446dbacaab5a5a98f640c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:44.855083 15887 consensus_queue.cc:1035] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [LEADER]: Connected to new peer: Peer: permanent_uuid: "324319386da6425eafce632cdd225399" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:44.855294 15891 consensus_queue.cc:1035] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f190e88725a442f4a37a5fd9a47a4508" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 35939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:44.861510 15894 mvcc.cc:205] Tried to move back new op lower bound from 7334031543716696064 to 7334031543631843328. Current Snapshot: MvccSnapshot[applied={T|T < 7334031543716696064}]
I20260927 18:50:44.865200 15895 mvcc.cc:205] Tried to move back new op lower bound from 7334031543716696064 to 7334031543631843328. Current Snapshot: MvccSnapshot[applied={T|T < 7334031543716696064}]
I20260927 18:50:44.870963 15413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:44.871141 15413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:44.873513 15413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 890d8903e7cd40cf9d13d118ec6a31fb: Sending DeleteTablet for 3 replicas of tablet 98b13095eb4b4bd0a9a1c284dcc6302c
I20260927 18:50:44.874300 15536 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b13095eb4b4bd0a9a1c284dcc6302c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:44 UTC) from {username='slave'} at 127.0.0.1:54866
I20260927 18:50:44.874325 15667 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b13095eb4b4bd0a9a1c284dcc6302c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:44 UTC) from {username='slave'} at 127.0.0.1:38468
I20260927 18:50:44.874786 15798 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b13095eb4b4bd0a9a1c284dcc6302c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:44 UTC) from {username='slave'} at 127.0.0.1:38488
I20260927 18:50:44.875114 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 15474
I20260927 18:50:44.875180 15905 tablet_replica.cc:330] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c: stopping tablet replica
I20260927 18:50:44.875128 15904 tablet_replica.cc:330] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399: stopping tablet replica
I20260927 18:50:44.875492 15905 raft_consensus.cc:2229] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:50:44.875924 15905 raft_consensus.cc:2258] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:44.876544 15904 raft_consensus.cc:2229] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:44.876876 15904 raft_consensus.cc:2258] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:44.877463 15905 ts_tablet_manager.cc:1905] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:44.878326 15904 ts_tablet_manager.cc:1905] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:44.880259 15905 ts_tablet_manager.cc:1918] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:44.880353 15905 log.cc:1196] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/98b13095eb4b4bd0a9a1c284dcc6302c
I20260927 18:50:44.880664 15905 ts_tablet_manager.cc:1939] T 98b13095eb4b4bd0a9a1c284dcc6302c P f083633ee17446dbacaab5a5a98f640c: Deleting consensus metadata
I20260927 18:50:44.881126 15904 ts_tablet_manager.cc:1918] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:44.881215 15904 log.cc:1196] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/98b13095eb4b4bd0a9a1c284dcc6302c
I20260927 18:50:44.881533 15904 ts_tablet_manager.cc:1939] T 98b13095eb4b4bd0a9a1c284dcc6302c P 324319386da6425eafce632cdd225399: Deleting consensus metadata
I20260927 18:50:44.881820 15399 catalog_manager.cc:4841] TS f083633ee17446dbacaab5a5a98f640c (127.29.159.67:41861): tablet 98b13095eb4b4bd0a9a1c284dcc6302c (table test-table [id=f13e4ae8bdf547a892440595747d4b3c]) successfully deleted
I20260927 18:50:44.882527 15399 catalog_manager.cc:4841] TS 324319386da6425eafce632cdd225399 (127.29.159.66:35481): tablet 98b13095eb4b4bd0a9a1c284dcc6302c (table test-table [id=f13e4ae8bdf547a892440595747d4b3c]) successfully deleted
W20260927 18:50:44.886806 15398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:35939 (error 108)
I20260927 18:50:44.886965 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 15605
I20260927 18:50:44.887974 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:44.887631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:35939 (local address 127.0.0.1:54872)
0927 18:50:44.887791 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:44.887873 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:50:44.888718 15398 catalog_manager.cc:4584] TS f190e88725a442f4a37a5fd9a47a4508 (127.29.159.65:35939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98b13095eb4b4bd0a9a1c284dcc6302c: Network error: Client connection negotiation failed: client connection to 127.29.159.65:35939: connect: Connection refused (error 111)
I20260927 18:50:44.892845 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 15736
I20260927 18:50:44.898470 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 15383
2026-09-27T18:50:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1024 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:44 dist-test-slave-8tg2 krb5kdc[15909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:44 dist-test-slave-8tg2 krb5kdc[15909](info): set up 2 sockets
Sep 27 18:50:44 dist-test-slave-8tg2 krb5kdc[15909](info): commencing operation
krb5kdc: starting...
W20260927 18:50:45.931373 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:50:46 dist-test-slave-8tg2 krb5kdc[15909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:50:46.054019 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:33161
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:33161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:46.166178 15922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:46.166440 15922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:46.166479 15922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:46.170608 15922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:46.170665 15922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:46.170684 15922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:46.170701 15922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:46.170717 15922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:46.175999 15922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:33161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:33161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:46.177065 15922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:46.178117 15922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:46.186568 15930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:46.186568 15927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:46.186582 15928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:46.186774 15922 server_base.cc:1034] running on GCE node
I20260927 18:50:46.187558 15922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:46.188616 15922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:46.189787 15922 hybrid_clock.cc:648] HybridClock initialized: now 1790535046189758 us; error 43 us; skew 500 ppm
Sep 27 18:50:46 dist-test-slave-8tg2 krb5kdc[15909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535046, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:46.192494 15922 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:50:46.193358 15922 webserver.cc:460] Webserver started at http://127.29.159.126:43701/ using document root <none> and password file <none>
I20260927 18:50:46.193869 15922 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:46.193926 15922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:46.194095 15922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:46.196117 15922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "ff7acbe1576e45009d51f1db93f475ce"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "bbbc3b96c9e171c19a575b8ec37518fb"
server_key_iv: "dcfc04d31226ec9c29e6a8e95c4173d9"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.196624 15922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "ff7acbe1576e45009d51f1db93f475ce"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "bbbc3b96c9e171c19a575b8ec37518fb"
server_key_iv: "dcfc04d31226ec9c29e6a8e95c4173d9"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.200377 15922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:50:46.202802 15937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:46.204093 15922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:50:46.204239 15922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "ff7acbe1576e45009d51f1db93f475ce"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "bbbc3b96c9e171c19a575b8ec37518fb"
server_key_iv: "dcfc04d31226ec9c29e6a8e95c4173d9"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.204362 15922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:46.229212 15922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:46.232486 15922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:46.232700 15922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:46.239952 15922 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:33161
I20260927 18:50:46.239971 15989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:33161 every 8 connection(s)
I20260927 18:50:46.240954 15922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:46.243744 15990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:46.249188 15990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce: Bootstrap starting.
I20260927 18:50:46.250474 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 15922
I20260927 18:50:46.250607 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:50:46.250906 30333 external_mini_cluster.cc:1428] Setting key 919611bce3cb5bebb07d71a4e95f32d1
I20260927 18:50:46.252071 15990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:46.252910 15990 log.cc:824] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:46.254783 15990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce: No bootstrap required, opened a new log
I20260927 18:50:46.258276 15990 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7acbe1576e45009d51f1db93f475ce" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33161 } }
I20260927 18:50:46.258539 15990 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:46.258621 15990 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff7acbe1576e45009d51f1db93f475ce, State: Initialized, Role: FOLLOWER
I20260927 18:50:46.259089 15990 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7acbe1576e45009d51f1db93f475ce" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33161 } }
I20260927 18:50:46.259240 15990 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:46.259330 15990 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:46.259450 15990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [term 0 FOLLOWER]: Advancing to term 1
Sep 27 18:50:46 dist-test-slave-8tg2 krb5kdc[15909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:46.260390 15990 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7acbe1576e45009d51f1db93f475ce" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33161 } }
I20260927 18:50:46.260746 15990 leader_election.cc:302] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff7acbe1576e45009d51f1db93f475ce; no voters:
I20260927 18:50:46.261053 15990 leader_election.cc:288] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:46.261165 15995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:46.261380 15995 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [term 1 LEADER]: Becoming Leader. State: Replica: ff7acbe1576e45009d51f1db93f475ce, State: Running, Role: LEADER
I20260927 18:50:46.261720 15995 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7acbe1576e45009d51f1db93f475ce" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33161 } }
I20260927 18:50:46.262457 15990 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:46.262954 15995 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff7acbe1576e45009d51f1db93f475ce. Latest consensus state: current_term: 1 leader_uuid: "ff7acbe1576e45009d51f1db93f475ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7acbe1576e45009d51f1db93f475ce" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33161 } } }
I20260927 18:50:46.262980 15996 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff7acbe1576e45009d51f1db93f475ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7acbe1576e45009d51f1db93f475ce" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33161 } } }
I20260927 18:50:46.263087 15996 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:46.263087 15995 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:46.263502 16000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:46.266566 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:46.252123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.29.159.126:33161)
0927 18:50:46.252583 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:46.252601 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:46.252637 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:50:46.253406 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:46.253432 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:46.253832 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:46.254171 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:46.254969 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.255993 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.256846 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.257314 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.257483 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:46.260219 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:46.260224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:46.260236 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:46.260262 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:46.262326 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:46.262832 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:46.262842 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:46.262849 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:46.262994 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:46.263522 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:46.263528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:46.263530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:46.264094 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:46.264324 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:46.265172 (+ 848us) server_negotiation.cc:299] Negotiation successful
0927 18:50:46.265526 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":145,"threads_started":1}
I20260927 18:50:46.266980 16000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:46.272727 16000 catalog_manager.cc:1347] Generated new cluster ID: fcd03656534c49c587757cdd6df682be
I20260927 18:50:46.272806 16000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:46.299638 16000 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:46.300678 16000 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:46.306032 16000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce: Generated new TSK 0
I20260927 18:50:46.306792 16000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:50:46.346688 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33161
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:46.471963 16016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:46.472239 16016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:46.472283 16016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:46.476333 16016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:46.476401 16016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:46.476461 16016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:50:46.481801 16016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:46.483026 16016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:46.484164 16016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:46.494374 16021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:46.494392 16024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:46.494410 16022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:46.494871 16016 server_base.cc:1034] running on GCE node
I20260927 18:50:46.495244 16016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:46.495946 16016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:46.497126 16016 hybrid_clock.cc:648] HybridClock initialized: now 1790535046497108 us; error 27 us; skew 500 ppm
Sep 27 18:50:46 dist-test-slave-8tg2 krb5kdc[15909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535046, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:46.500033 16016 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:50:46.500896 16016 webserver.cc:460] Webserver started at http://127.29.159.65:41825/ using document root <none> and password file <none>
I20260927 18:50:46.501463 16016 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:46.501554 16016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:46.501786 16016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:46.504112 16016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "7c730b3307b045edb00fedb53ec3d58f"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "8f9a1ab757a367eca5c590a1d9bc8e5e"
server_key_iv: "73ce98ece1e08c5dc2cfb01ffdde27db"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.504683 16016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "7c730b3307b045edb00fedb53ec3d58f"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "8f9a1ab757a367eca5c590a1d9bc8e5e"
server_key_iv: "73ce98ece1e08c5dc2cfb01ffdde27db"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.508466 16016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:50:46.511005 16031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:46.512331 16016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:46.512468 16016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "7c730b3307b045edb00fedb53ec3d58f"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "8f9a1ab757a367eca5c590a1d9bc8e5e"
server_key_iv: "73ce98ece1e08c5dc2cfb01ffdde27db"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.512593 16016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:46.522804 16016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:46.525549 16016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:46.525751 16016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:46.526310 16016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:46.527153 16016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:46.527223 16016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:46.527292 16016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:46.527347 16016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:46.536342 16016 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:38843
I20260927 18:50:46.536361 16144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:38843 every 8 connection(s)
I20260927 18:50:46.537204 16016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:50:46.543819 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 16016
I20260927 18:50:46.543951 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:50:46.544241 30333 external_mini_cluster.cc:1428] Setting key a5b0309d7d894dc68fefba8bf396a474
Sep 27 18:50:46 dist-test-slave-8tg2 krb5kdc[15909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535046, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:46.551308 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:46.539039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:54883 (local address 127.29.159.126:33161)
0927 18:50:46.539186 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:46.539191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:46.540105 (+ 914us) server_negotiation.cc:407] Connection header received
0927 18:50:46.541336 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:46.541339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:46.541386 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:46.541462 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:46.542561 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.543082 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.544367 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.544692 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.544813 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:46.547584 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:46.547587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:46.547589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:46.547611 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:46.549053 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:46.549680 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:46.549683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:46.549684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:46.549755 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:46.550147 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:46.550149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:46.550150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:46.550309 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:46.550380 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:46.551045 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 18:50:46.551183 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:50:46.552472 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:46.539517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:33161 (local address 127.29.159.65:54883)
0927 18:50:46.539950 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:46.539999 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:46.541139 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:46.541617 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:46.541624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:46.541954 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:46.542429 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:46.542442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.543183 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.543186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:46.544229 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:46.544238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.544849 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.544853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:46.545016 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:46.545641 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:46.545656 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:46.547406 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:46.549170 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:46.549180 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:46.549197 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:46.549568 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:46.549854 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:46.549860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:46.549865 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:46.550049 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:46.550415 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:46.550423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:46.550753 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 18:50:46.550989 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 18:50:46.551370 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260927 18:50:46.553753 16145 heartbeater.cc:347] Connected to a master server at 127.29.159.126:33161
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
I20260927 18:50:46.553998 16145 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:46.554461 16145 heartbeater.cc:511] Master 127.29.159.126:33161 requested a full tablet report, sending...
I20260927 18:50:46.556018 15954 ts_manager.cc:194] Registered new tserver with Master: 7c730b3307b045edb00fedb53ec3d58f (127.29.159.65:38843)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
I20260927 18:50:46.557355 15954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:54883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:50:46.574008 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33161
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:46.683229 16150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:46.683514 16150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:46.683559 16150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:46.687527 16150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:46.687585 16150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:46.687680 16150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:50:46.692945 16150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:46.693960 16150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:46.694981 16150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:46.705083 16155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:46.705098 16158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:46.705170 16150 server_base.cc:1034] running on GCE node
W20260927 18:50:46.705103 16156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:46.705721 16150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:46.706393 16150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:46.707574 16150 hybrid_clock.cc:648] HybridClock initialized: now 1790535046707545 us; error 39 us; skew 500 ppm
Sep 27 18:50:46 dist-test-slave-8tg2 krb5kdc[15909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535046, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:46.714819 16150 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:50:46.716094 16150 webserver.cc:460] Webserver started at http://127.29.159.66:33327/ using document root <none> and password file <none>
I20260927 18:50:46.716991 16150 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:46.717093 16150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:46.717393 16150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:46.720685 16150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "0c1a39459e9e4774a07adf73c120c8a0"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "936b4eff1327dbbb9e6b7ed15e3c0517"
server_key_iv: "7fae183486f4d8313f7aa502010c4d73"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.721460 16150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "0c1a39459e9e4774a07adf73c120c8a0"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "936b4eff1327dbbb9e6b7ed15e3c0517"
server_key_iv: "7fae183486f4d8313f7aa502010c4d73"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.726742 16150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:50:46.729274 16165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:46.730547 16150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 18:50:46.730692 16150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "0c1a39459e9e4774a07adf73c120c8a0"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "936b4eff1327dbbb9e6b7ed15e3c0517"
server_key_iv: "7fae183486f4d8313f7aa502010c4d73"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.730819 16150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:46.744293 16150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:46.747507 16150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:46.747689 16150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:46.748242 16150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:46.749010 16150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:46.749078 16150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:46.749147 16150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:46.749203 16150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:46.757784 16150 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:42251
I20260927 18:50:46.757804 16278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:42251 every 8 connection(s)
I20260927 18:50:46.758620 16150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:50:46.760020 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 16150
I20260927 18:50:46.760154 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:50:46.760476 30333 external_mini_cluster.cc:1428] Setting key b94164d5390df191b44154fb74162f3d
Sep 27 18:50:46 dist-test-slave-8tg2 krb5kdc[15909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535046, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
I20260927 18:50:46.773573 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:46.760453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:37939 (local address 127.29.159.126:33161)
0927 18:50:46.760580 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:46.760585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:46.761536 (+ 951us) server_negotiation.cc:407] Connection header received
0927 18:50:46.762904 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:46.762910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:46.762972 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:46.763066 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:46.764454 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.765068 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.766851 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.767098 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.767184 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:46.770020 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:46.770024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:46.770026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:46.770049 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:46.771583 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:46.772084 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:46.772089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:46.772091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:46.772173 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:46.772498 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:46.772510 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:46.772512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:46.772686 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:46.772792 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:46.773273 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 18:50:46.773388 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:50:46.774365 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:46.760934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:33161 (local address 127.29.159.66:37939)
0927 18:50:46.761382 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:46.761431 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:46.762706 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:46.763249 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:46.763255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:46.763767 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:46.764298 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:46.764314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.765193 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.765199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:46.766694 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:46.766708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.767258 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.767265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:46.767518 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:46.768155 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:46.768184 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:46.769847 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:46.771704 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:46.771709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:46.771724 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:46.771970 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:46.772263 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:46.772268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:46.772270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:46.772404 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:46.772789 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:46.772795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:46.773045 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 18:50:46.773231 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 18:50:46.773535 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":127,"threads_started":1}
I20260927 18:50:46.775741 16279 heartbeater.cc:347] Connected to a master server at 127.29.159.126:33161
I20260927 18:50:46.776033 16279 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:46.776628 16279 heartbeater.cc:511] Master 127.29.159.126:33161 requested a full tablet report, sending...
I20260927 18:50:46.777740 15954 ts_manager.cc:194] Registered new tserver with Master: 0c1a39459e9e4774a07adf73c120c8a0 (127.29.159.66:42251)
I20260927 18:50:46.778625 15954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:37939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:50:46.789821 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33161
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:46.900893 16284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:46.901173 16284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:46.901221 16284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:46.905296 16284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:46.905356 16284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:46.905417 16284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:50:46.910909 16284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:46.911960 16284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:46.912933 16284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:46.922649 16290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:46.922660 16289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:46.922657 16292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:46.922928 16284 server_base.cc:1034] running on GCE node
I20260927 18:50:46.923336 16284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:46.923956 16284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:46.925119 16284 hybrid_clock.cc:648] HybridClock initialized: now 1790535046925106 us; error 27 us; skew 500 ppm
Sep 27 18:50:46 dist-test-slave-8tg2 krb5kdc[15909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535046, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:46.927732 16284 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:50:46.928576 16284 webserver.cc:460] Webserver started at http://127.29.159.67:36733/ using document root <none> and password file <none>
I20260927 18:50:46.929139 16284 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:46.929229 16284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:46.929459 16284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:46.931411 16284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "325aeb1e7e6841a9b0f12d98d0503f30"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "b4dd2ddea41935f2973f70c59cbf4453"
server_key_iv: "8662612abe6bda6845dd7e819b6ed69a"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.932035 16284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "325aeb1e7e6841a9b0f12d98d0503f30"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "b4dd2ddea41935f2973f70c59cbf4453"
server_key_iv: "8662612abe6bda6845dd7e819b6ed69a"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.935968 16284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 18:50:46.938558 16299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:46.939965 16284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:50:46.940114 16284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "325aeb1e7e6841a9b0f12d98d0503f30"
format_stamp: "Formatted at 2026-09-27 18:50:46 on dist-test-slave-8tg2"
server_key: "b4dd2ddea41935f2973f70c59cbf4453"
server_key_iv: "8662612abe6bda6845dd7e819b6ed69a"
server_key_version: "encryptionkey@0"
I20260927 18:50:46.940225 16284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:46.952629 16284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:46.955823 16284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:46.956049 16284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:46.956599 16284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:46.957382 16284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:46.957448 16284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:46.957517 16284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:46.957568 16284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:46.966432 16284 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:46847
I20260927 18:50:46.966497 16412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:46847 every 8 connection(s)
I20260927 18:50:46.967409 16284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:50:46.976070 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 16284
I20260927 18:50:46.976207 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:50:46.976513 30333 external_mini_cluster.cc:1428] Setting key 9ef707f48e331fd8bd155aefb6956e79
Sep 27 18:50:46 dist-test-slave-8tg2 krb5kdc[15909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535046, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:46.982050 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:46.969295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:36493 (local address 127.29.159.126:33161)
0927 18:50:46.969510 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:46.969515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:46.970273 (+ 758us) server_negotiation.cc:407] Connection header received
0927 18:50:46.971683 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:46.971687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:46.971744 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:46.971887 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:46.973076 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.973612 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.974769 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.974996 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.975108 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:46.978120 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:46.978124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:46.978127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:46.978162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:46.980006 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:46.980572 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:46.980575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:46.980576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:46.980659 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:46.980966 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:46.980968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:46.980969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:46.981158 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:46.981241 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:46.981796 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 18:50:46.981925 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 18:50:46.983000 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:46.969693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:33161 (local address 127.29.159.67:36493)
0927 18:50:46.970126 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:46.970168 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:46.971408 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:46.972071 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:46.972079 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:46.972435 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:46.972924 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:46.972937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.973741 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.973744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:46.974641 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:46.974650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.975149 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.975154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:46.975329 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:46.976042 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:46.976060 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:46.977960 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:46.980115 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:46.980121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:46.980134 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:46.980477 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:46.980741 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:46.980743 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:46.980745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:46.980884 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:46.981263 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:46.981269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:46.981539 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 18:50:46.981709 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 18:50:46.982009 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260927 18:50:46.984282 16413 heartbeater.cc:347] Connected to a master server at 127.29.159.126:33161
I20260927 18:50:46.984565 16413 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:46.985127 16413 heartbeater.cc:511] Master 127.29.159.126:33161 requested a full tablet report, sending...
I20260927 18:50:46.986258 15954 ts_manager.cc:194] Registered new tserver with Master: 325aeb1e7e6841a9b0f12d98d0503f30 (127.29.159.67:46847)
I20260927 18:50:46.986835 15954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:36493
I20260927 18:50:46.990808 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:50:46.999976 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:46.992789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.29.159.126:33161)
0927 18:50:46.992919 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:46.992923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:46.993124 (+ 201us) server_negotiation.cc:407] Connection header received
0927 18:50:46.993228 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:46.993230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:46.993265 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:46.993325 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:46.994005 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.994508 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.995326 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:46.995594 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:46.995669 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:46.997071 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:46.997074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:46.997076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:46.997098 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:46.998372 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:46.998842 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:46.998844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:46.998846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:46.998917 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:46.999182 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:46.999184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:46.999185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:46.999372 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:46.999579 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:46.999779 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 18:50:46.999857 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:50:47.003023 15954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:50:47.006459 15954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:50:47.022807 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.017262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.29.159.65:38843)
0927 18:50:47.017946 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:47.017963 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:47.017984 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:50:47.018059 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:47.018064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:47.018251 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:47.018426 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:47.018668 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.019782 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.021130 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.022005 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.022234 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.022357 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:47.022535 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 18:50:47.022643 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":89,"threads_started":1}
I20260927 18:50:47.022811 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.016863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:38843 (local address 127.0.0.1:36342)
0927 18:50:47.017600 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:47.017614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:47.017695 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:47.018425 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:47.018427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:47.018448 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:47.018543 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.018549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.019902 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.019906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.021012 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.021022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.022134 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.022137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.022293 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.022339 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 18:50:47.022487 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 18:50:47.022633 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":23,"thread_start_us":97,"threads_started":1}
I20260927 18:50:47.022802 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.016064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:42251 (local address 127.0.0.1:39192)
0927 18:50:47.017017 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:47.017029 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:47.017226 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:47.018080 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:47.018083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:47.018100 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:47.018187 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.018192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.019660 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.019663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.021097 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.021108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.022316 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.022320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.022422 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.022496 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 18:50:47.022575 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:50:47.022652 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":74,"spinlock_wait_cycles":4480,"thread_start_us":136,"threads_started":1}
I20260927 18:50:47.023283 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.016048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:46847 (local address 127.0.0.1:59490)
0927 18:50:47.016906 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:47.016957 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:47.017086 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:47.017900 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:47.017910 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:47.017938 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:47.018089 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.018101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.019469 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.019473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.021235 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.021243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.022100 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.022104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.022884 (+ 780us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.022924 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:50:47.023024 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:50:47.023141 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":239,"thread_start_us":104,"threads_started":1}
I20260927 18:50:47.023307 16426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.016229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.29.159.67:46847)
0927 18:50:47.017100 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:47.017113 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:47.017141 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:50:47.017228 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:47.017235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:47.017484 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:47.017693 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:47.018246 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.019309 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.021359 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.021964 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.022101 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.022263 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:47.023052 (+ 789us) server_negotiation.cc:299] Negotiation successful
0927 18:50:47.023158 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":78,"threads_started":1}
I20260927 18:50:47.025964 16347 tablet_service.cc:1467] Processing CreateTablet for tablet 7cfc9408f4a4471680c3ca986a8aba92 (DEFAULT_TABLE table=test-table [id=43d016164cda4717acb848ea66f29de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:47.025956 16079 tablet_service.cc:1467] Processing CreateTablet for tablet 7cfc9408f4a4471680c3ca986a8aba92 (DEFAULT_TABLE table=test-table [id=43d016164cda4717acb848ea66f29de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:47.026510 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.016501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.29.159.66:42251)
0927 18:50:47.016874 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:47.016886 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:47.017162 (+ 276us) server_negotiation.cc:407] Connection header received
0927 18:50:47.017369 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:47.017375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:47.017777 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:47.017920 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:47.018338 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.019532 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.021214 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.021993 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.022110 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.022260 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:47.022636 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 18:50:47.022751 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260927 18:50:47.026979 16347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cfc9408f4a4471680c3ca986a8aba92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:47.027017 16079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cfc9408f4a4471680c3ca986a8aba92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:47.029208 16213 tablet_service.cc:1467] Processing CreateTablet for tablet 7cfc9408f4a4471680c3ca986a8aba92 (DEFAULT_TABLE table=test-table [id=43d016164cda4717acb848ea66f29de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:47.030221 16213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cfc9408f4a4471680c3ca986a8aba92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:47.033895 16430 tablet_bootstrap.cc:492] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f: Bootstrap starting.
I20260927 18:50:47.035794 16431 tablet_bootstrap.cc:492] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: Bootstrap starting.
I20260927 18:50:47.036579 16430 tablet_bootstrap.cc:654] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:47.036746 16432 tablet_bootstrap.cc:492] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0: Bootstrap starting.
I20260927 18:50:47.038234 16431 tablet_bootstrap.cc:654] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:47.038707 16432 tablet_bootstrap.cc:654] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:47.038714 16430 log.cc:824] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:47.039613 16431 log.cc:824] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:47.039909 16432 log.cc:824] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:47.041278 16430 tablet_bootstrap.cc:492] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f: No bootstrap required, opened a new log
I20260927 18:50:47.041509 16431 tablet_bootstrap.cc:492] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: No bootstrap required, opened a new log
I20260927 18:50:47.041507 16430 ts_tablet_manager.cc:1397] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 18:50:47.041702 16431 ts_tablet_manager.cc:1397] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260927 18:50:47.041893 16432 tablet_bootstrap.cc:492] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0: No bootstrap required, opened a new log
I20260927 18:50:47.042070 16432 ts_tablet_manager.cc:1397] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:50:47.044844 16431 raft_consensus.cc:348] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 46847 } } peers { permanent_uuid: "0c1a39459e9e4774a07adf73c120c8a0" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 42251 } } peers { permanent_uuid: "7c730b3307b045edb00fedb53ec3d58f" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38843 } }
I20260927 18:50:47.045117 16431 raft_consensus.cc:374] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:47.045171 16431 raft_consensus.cc:729] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325aeb1e7e6841a9b0f12d98d0503f30, State: Initialized, Role: FOLLOWER
I20260927 18:50:47.045619 16431 consensus_queue.cc:260] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 46847 } } peers { permanent_uuid: "0c1a39459e9e4774a07adf73c120c8a0" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 42251 } } peers { permanent_uuid: "7c730b3307b045edb00fedb53ec3d58f" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38843 } }
I20260927 18:50:47.046304 16413 heartbeater.cc:503] Master 127.29.159.126:33161 was elected leader, sending a full tablet report...
I20260927 18:50:47.046167 16432 raft_consensus.cc:348] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 46847 } } peers { permanent_uuid: "0c1a39459e9e4774a07adf73c120c8a0" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 42251 } } peers { permanent_uuid: "7c730b3307b045edb00fedb53ec3d58f" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38843 } }
I20260927 18:50:47.046428 16432 raft_consensus.cc:374] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:47.046496 16432 raft_consensus.cc:729] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1a39459e9e4774a07adf73c120c8a0, State: Initialized, Role: FOLLOWER
I20260927 18:50:47.046672 16431 ts_tablet_manager.cc:1428] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 18:50:47.046939 16432 consensus_queue.cc:260] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 46847 } } peers { permanent_uuid: "0c1a39459e9e4774a07adf73c120c8a0" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 42251 } } peers { permanent_uuid: "7c730b3307b045edb00fedb53ec3d58f" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38843 } }
I20260927 18:50:47.047636 16279 heartbeater.cc:503] Master 127.29.159.126:33161 was elected leader, sending a full tablet report...
I20260927 18:50:47.047645 16430 raft_consensus.cc:348] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 46847 } } peers { permanent_uuid: "0c1a39459e9e4774a07adf73c120c8a0" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 42251 } } peers { permanent_uuid: "7c730b3307b045edb00fedb53ec3d58f" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38843 } }
I20260927 18:50:47.047914 16430 raft_consensus.cc:374] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:47.047977 16430 raft_consensus.cc:729] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c730b3307b045edb00fedb53ec3d58f, State: Initialized, Role: FOLLOWER
I20260927 18:50:47.048401 16430 consensus_queue.cc:260] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 46847 } } peers { permanent_uuid: "0c1a39459e9e4774a07adf73c120c8a0" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 42251 } } peers { permanent_uuid: "7c730b3307b045edb00fedb53ec3d58f" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38843 } }
I20260927 18:50:47.049108 16145 heartbeater.cc:503] Master 127.29.159.126:33161 was elected leader, sending a full tablet report...
I20260927 18:50:47.049547 16430 ts_tablet_manager.cc:1428] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 18:50:47.050190 16432 ts_tablet_manager.cc:1428] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 18:50:47.194885 16438 raft_consensus.cc:482] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:47.195139 16438 raft_consensus.cc:504] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 46847 } } peers { permanent_uuid: "0c1a39459e9e4774a07adf73c120c8a0" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 42251 } } peers { permanent_uuid: "7c730b3307b045edb00fedb53ec3d58f" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38843 } }
I20260927 18:50:47.196518 16438 leader_election.cc:288] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 325aeb1e7e6841a9b0f12d98d0503f30 (127.29.159.67:46847), 0c1a39459e9e4774a07adf73c120c8a0 (127.29.159.66:42251)
I20260927 18:50:47.202133 16426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.196836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:59723 (local address 127.29.159.67:46847)
0927 18:50:47.196970 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:47.196974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:47.197133 (+ 159us) server_negotiation.cc:407] Connection header received
0927 18:50:47.197302 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:47.197305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:47.197367 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:47.197448 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:47.197922 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.198714 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.200695 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.201306 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.201432 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.201491 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:47.201919 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 18:50:47.202004 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:50:47.202133 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.196902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:54441 (local address 127.29.159.66:42251)
0927 18:50:47.197048 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:47.197052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:47.197246 (+ 194us) server_negotiation.cc:407] Connection header received
0927 18:50:47.197323 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:47.197326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:47.197374 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:47.197449 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:47.197916 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.198714 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.200693 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.201306 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.201432 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.201484 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:47.201919 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 18:50:47.202016 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:50:47.202153 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.196824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:42251 (local address 127.29.159.65:54441)
0927 18:50:47.197138 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:47.197149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:47.197239 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:47.197482 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:47.197485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:47.197528 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:47.197783 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.197790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.198848 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.198852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.200563 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.200575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.201486 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.201490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.201702 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.201751 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 18:50:47.201840 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:50:47.201939 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":303,"spinlock_wait_cycles":29312,"thread_start_us":126,"threads_started":1}
I20260927 18:50:47.202152 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.196699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:46847 (local address 127.29.159.65:59723)
0927 18:50:47.197037 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:47.197052 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:47.197169 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:47.197482 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:47.197485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:47.197529 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:47.197783 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.197790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.198848 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.198852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.200563 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.200575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.201486 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.201490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.201706 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.201757 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 18:50:47.201840 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 18:50:47.201941 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":335,"spinlock_wait_cycles":10496,"thread_start_us":86,"threads_started":1}
I20260927 18:50:47.202925 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfc9408f4a4471680c3ca986a8aba92" candidate_uuid: "7c730b3307b045edb00fedb53ec3d58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" is_pre_election: true
I20260927 18:50:47.202955 16233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfc9408f4a4471680c3ca986a8aba92" candidate_uuid: "7c730b3307b045edb00fedb53ec3d58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1a39459e9e4774a07adf73c120c8a0" is_pre_election: true
I20260927 18:50:47.203244 16233 raft_consensus.cc:2454] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c730b3307b045edb00fedb53ec3d58f in term 0.
I20260927 18:50:47.203249 16367 raft_consensus.cc:2454] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c730b3307b045edb00fedb53ec3d58f in term 0.
I20260927 18:50:47.203765 16034 leader_election.cc:302] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c1a39459e9e4774a07adf73c120c8a0, 325aeb1e7e6841a9b0f12d98d0503f30, 7c730b3307b045edb00fedb53ec3d58f; no voters:
I20260927 18:50:47.204123 16438 raft_consensus.cc:2781] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:50:47.204206 16438 raft_consensus.cc:482] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:50:47.204237 16438 raft_consensus.cc:3037] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:47.205521 16438 raft_consensus.cc:504] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 46847 } } peers { permanent_uuid: "0c1a39459e9e4774a07adf73c120c8a0" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 42251 } } peers { permanent_uuid: "7c730b3307b045edb00fedb53ec3d58f" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38843 } }
I20260927 18:50:47.205950 16438 leader_election.cc:288] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [CANDIDATE]: Term 1 election: Requested vote from peers 325aeb1e7e6841a9b0f12d98d0503f30 (127.29.159.67:46847), 0c1a39459e9e4774a07adf73c120c8a0 (127.29.159.66:42251)
I20260927 18:50:47.206348 16233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfc9408f4a4471680c3ca986a8aba92" candidate_uuid: "7c730b3307b045edb00fedb53ec3d58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1a39459e9e4774a07adf73c120c8a0"
I20260927 18:50:47.206359 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfc9408f4a4471680c3ca986a8aba92" candidate_uuid: "7c730b3307b045edb00fedb53ec3d58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "325aeb1e7e6841a9b0f12d98d0503f30"
I20260927 18:50:47.206459 16233 raft_consensus.cc:3037] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:47.206560 16367 raft_consensus.cc:3037] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:47.207659 16233 raft_consensus.cc:2454] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c730b3307b045edb00fedb53ec3d58f in term 1.
I20260927 18:50:47.207993 16367 raft_consensus.cc:2454] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c730b3307b045edb00fedb53ec3d58f in term 1.
I20260927 18:50:47.208045 16034 leader_election.cc:302] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1a39459e9e4774a07adf73c120c8a0, 7c730b3307b045edb00fedb53ec3d58f; no voters:
I20260927 18:50:47.208304 16438 raft_consensus.cc:2781] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:47.208575 16438 raft_consensus.cc:686] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 1 LEADER]: Becoming Leader. State: Replica: 7c730b3307b045edb00fedb53ec3d58f, State: Running, Role: LEADER
I20260927 18:50:47.208986 16438 consensus_queue.cc:237] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 46847 } } peers { permanent_uuid: "0c1a39459e9e4774a07adf73c120c8a0" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 42251 } } peers { permanent_uuid: "7c730b3307b045edb00fedb53ec3d58f" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38843 } }
I20260927 18:50:47.212419 15954 catalog_manager.cc:5495] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c730b3307b045edb00fedb53ec3d58f (127.29.159.65). New cstate: current_term: 1 leader_uuid: "7c730b3307b045edb00fedb53ec3d58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 46847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c1a39459e9e4774a07adf73c120c8a0" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c730b3307b045edb00fedb53ec3d58f" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38843 } health_report { overall_health: HEALTHY } } }
W20260927 18:50:47.218420 16414 tablet.cc:2369] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:50:47.259035 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.255527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.29.159.65:38843)
0927 18:50:47.255676 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:47.255680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:47.255701 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:50:47.255765 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:47.255767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:47.255821 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:47.255938 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:47.256194 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.256698 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.257567 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.257786 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.257898 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.258204 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:47.258608 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:47.258769 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:47.258857 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 18:50:47.258897 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 18:50:47.259717 16280 tablet.cc:2369] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:50:47.265450 16367 raft_consensus.cc:1261] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30 [term 1 FOLLOWER]: Refusing update from remote peer 7c730b3307b045edb00fedb53ec3d58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:47.265450 16233 raft_consensus.cc:1261] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0 [term 1 FOLLOWER]: Refusing update from remote peer 7c730b3307b045edb00fedb53ec3d58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:47.266215 16438 consensus_queue.cc:1035] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "325aeb1e7e6841a9b0f12d98d0503f30" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:47.266403 16441 consensus_queue.cc:1035] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1a39459e9e4774a07adf73c120c8a0" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:47.274681 16445 mvcc.cc:205] Tried to move back new op lower bound from 7334031553591586816 to 7334031553371123712. Current Snapshot: MvccSnapshot[applied={T|T < 7334031553591586816}]
I20260927 18:50:47.283214 15954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:47.283411 15954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:47.285861 15954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff7acbe1576e45009d51f1db93f475ce: Sending DeleteTablet for 3 replicas of tablet 7cfc9408f4a4471680c3ca986a8aba92
I20260927 18:50:47.286734 16213 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cfc9408f4a4471680c3ca986a8aba92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:47 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:39192
I20260927 18:50:47.286818 16347 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cfc9408f4a4471680c3ca986a8aba92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:47 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:59490
I20260927 18:50:47.287261 16079 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cfc9408f4a4471680c3ca986a8aba92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:47 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:36342
I20260927 18:50:47.287304 16455 tablet_replica.cc:330] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: stopping tablet replica
I20260927 18:50:47.287698 16456 tablet_replica.cc:330] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f: stopping tablet replica
I20260927 18:50:47.287693 16455 raft_consensus.cc:2229] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:47.288007 16455 raft_consensus.cc:2258] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:47.288026 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 16016
I20260927 18:50:47.288023 16456 raft_consensus.cc:2229] T 7cfc9408f4a4471680c3ca986a8aba92 P 7c730b3307b045edb00fedb53ec3d58f [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:50:47.290345 16455 ts_tablet_manager.cc:1905] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:47.292706 16455 ts_tablet_manager.cc:1918] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:47.292789 16455 log.cc:1196] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/7cfc9408f4a4471680c3ca986a8aba92
I20260927 18:50:47.293037 16454 tablet_replica.cc:330] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0: stopping tablet replica
I20260927 18:50:47.293215 16455 ts_tablet_manager.cc:1939] T 7cfc9408f4a4471680c3ca986a8aba92 P 325aeb1e7e6841a9b0f12d98d0503f30: Deleting consensus metadata
I20260927 18:50:47.293402 16454 raft_consensus.cc:2229] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:47.293736 16454 raft_consensus.cc:2258] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:47.294498 15939 catalog_manager.cc:4841] TS 325aeb1e7e6841a9b0f12d98d0503f30 (127.29.159.67:46847): tablet 7cfc9408f4a4471680c3ca986a8aba92 (table test-table [id=43d016164cda4717acb848ea66f29de0]) successfully deleted
I20260927 18:50:47.295405 16454 ts_tablet_manager.cc:1905] T 7cfc9408f4a4471680c3ca986a8aba92 P 0c1a39459e9e4774a07adf73c120c8a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:47.295537 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 16150
W20260927 18:50:47.295545 15939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:38843 (error 108)
I20260927 18:50:47.297050 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.296703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:38843 (local address 127.0.0.1:36366)
0927 18:50:47.296852 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:47.296938 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 18:50:47.297412 15939 catalog_manager.cc:4584] TS 7c730b3307b045edb00fedb53ec3d58f (127.29.159.65:38843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7cfc9408f4a4471680c3ca986a8aba92: Network error: Client connection negotiation failed: client connection to 127.29.159.65:38843: connect: Connection refused (error 111)
I20260927 18:50:47.301362 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 16284
I20260927 18:50:47.302942 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.301512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:42251 (local address 127.0.0.1:39202)
0927 18:50:47.302824 (+ 1312us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:47.302873 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.66:42251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1238}
I20260927 18:50:47.306977 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 15922
2026-09-27T18:50:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2409 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T18:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:47Z Disabled control of system clock
I20260927 18:50:47.337467 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:34005
--webserver_interface=127.29.159.126
--webserver_port=0
--builtin_ntp_servers=127.29.159.84:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:50:47.452729 16461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:47.452999 16461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:47.453037 16461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:47.456915 16461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:47.456960 16461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:47.456980 16461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:47.456997 16461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:47.457011 16461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:47.461853 16461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:34005
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:34005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:47.463186 16461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:47.464623 16461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:47.472985 16467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:47.473002 16469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:47.473022 16466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:47.473372 16461 server_base.cc:1034] running on GCE node
I20260927 18:50:47.473888 16461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:47.474995 16461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:47.476172 16461 hybrid_clock.cc:648] HybridClock initialized: now 1790535047476125 us; error 63 us; skew 500 ppm
I20260927 18:50:47.477949 16461 webserver.cc:460] Webserver started at http://127.29.159.126:46461/ using document root <none> and password file <none>
I20260927 18:50:47.478554 16461 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:47.478645 16461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:47.478874 16461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:47.481040 16461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "6bcad2e56c4e4edfa958d71a0ae6b974"
format_stamp: "Formatted at 2026-09-27 18:50:47 on dist-test-slave-8tg2"
server_key: "cecbd54d7cbdbbe30d8f2a0d815e5fb1"
server_key_iv: "a7d925f24accc9502f818a93d5226b4f"
server_key_version: "encryptionkey@0"
I20260927 18:50:47.481643 16461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "6bcad2e56c4e4edfa958d71a0ae6b974"
format_stamp: "Formatted at 2026-09-27 18:50:47 on dist-test-slave-8tg2"
server_key: "cecbd54d7cbdbbe30d8f2a0d815e5fb1"
server_key_iv: "a7d925f24accc9502f818a93d5226b4f"
server_key_version: "encryptionkey@0"
I20260927 18:50:47.485340 16461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:50:47.487800 16475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:47.488984 16461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:47.489122 16461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "6bcad2e56c4e4edfa958d71a0ae6b974"
format_stamp: "Formatted at 2026-09-27 18:50:47 on dist-test-slave-8tg2"
server_key: "cecbd54d7cbdbbe30d8f2a0d815e5fb1"
server_key_iv: "a7d925f24accc9502f818a93d5226b4f"
server_key_version: "encryptionkey@0"
I20260927 18:50:47.489229 16461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:47.517221 16461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:47.518013 16461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:47.518240 16461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:47.525871 16461 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:34005
I20260927 18:50:47.525946 16527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:34005 every 8 connection(s)
I20260927 18:50:47.526960 16461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:47.530211 16528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:47.533401 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 16461
I20260927 18:50:47.533555 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:50:47.533843 30333 external_mini_cluster.cc:1428] Setting key e4e1ff67569791c927a50027ab74759b
I20260927 18:50:47.536274 16528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974: Bootstrap starting.
I20260927 18:50:47.538676 16528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:47.539680 16528 log.cc:824] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:47.541885 16528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974: No bootstrap required, opened a new log
I20260927 18:50:47.543536 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.535299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.29.159.126:34005)
0927 18:50:47.535994 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:47.536010 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:47.536047 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:50:47.536723 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:47.536741 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:47.537160 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:47.537446 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:47.537793 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.538824 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.540088 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.540703 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.540842 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.541676 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:47.541681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:47.541694 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:47.542138 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:47.542193 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:47.542212 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:47.542287 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:47.542443 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 18:50:47.542727 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":191,"threads_started":1}
I20260927 18:50:47.545882 16528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcad2e56c4e4edfa958d71a0ae6b974" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34005 } }
I20260927 18:50:47.546084 16528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:47.546173 16528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bcad2e56c4e4edfa958d71a0ae6b974, State: Initialized, Role: FOLLOWER
I20260927 18:50:47.546567 16528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcad2e56c4e4edfa958d71a0ae6b974" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34005 } }
I20260927 18:50:47.546707 16528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:47.546794 16528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:47.546909 16528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:47.547927 16528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcad2e56c4e4edfa958d71a0ae6b974" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34005 } }
I20260927 18:50:47.548251 16528 leader_election.cc:302] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bcad2e56c4e4edfa958d71a0ae6b974; no voters:
I20260927 18:50:47.548564 16528 leader_election.cc:288] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:47.548722 16533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:47.548962 16533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [term 1 LEADER]: Becoming Leader. State: Replica: 6bcad2e56c4e4edfa958d71a0ae6b974, State: Running, Role: LEADER
I20260927 18:50:47.549274 16533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcad2e56c4e4edfa958d71a0ae6b974" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34005 } }
I20260927 18:50:47.549753 16528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:47.550871 16534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bcad2e56c4e4edfa958d71a0ae6b974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcad2e56c4e4edfa958d71a0ae6b974" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34005 } } }
I20260927 18:50:47.551009 16534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:47.551271 16535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bcad2e56c4e4edfa958d71a0ae6b974. Latest consensus state: current_term: 1 leader_uuid: "6bcad2e56c4e4edfa958d71a0ae6b974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcad2e56c4e4edfa958d71a0ae6b974" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34005 } } }
I20260927 18:50:47.551373 16535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:47.551473 16542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:47.554733 16542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:47.560376 16542 catalog_manager.cc:1347] Generated new cluster ID: c74da1add9af4d45ac4761c21d5d57c9
I20260927 18:50:47.560458 16542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:47.572935 16542 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:47.573837 16542 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:47.577476 16542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974: Generated new TSK 0
I20260927 18:50:47.578140 16542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:47.587738 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34005
--builtin_ntp_servers=127.29.159.84:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:50:47.700985 16552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:47.701238 16552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:47.701282 16552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:47.705317 16552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:47.705371 16552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:47.705430 16552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:50:47.710059 16552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:47.711068 16552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:47.712598 16552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:47.722172 16558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:47.722218 16560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:47.722216 16552 server_base.cc:1034] running on GCE node
W20260927 18:50:47.722177 16557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:47.722718 16552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:47.723465 16552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:47.724665 16552 hybrid_clock.cc:648] HybridClock initialized: now 1790535047724634 us; error 32 us; skew 500 ppm
I20260927 18:50:47.726696 16552 webserver.cc:460] Webserver started at http://127.29.159.65:38325/ using document root <none> and password file <none>
I20260927 18:50:47.727321 16552 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:47.727449 16552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:47.727692 16552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:47.729701 16552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "58d5656ab80340558b106131c9bcf7c1"
format_stamp: "Formatted at 2026-09-27 18:50:47 on dist-test-slave-8tg2"
server_key: "4c59d28d5bcf728396ef2fab1b17938f"
server_key_iv: "1965b538c480be5ba9117a3540437d39"
server_key_version: "encryptionkey@0"
I20260927 18:50:47.730289 16552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "58d5656ab80340558b106131c9bcf7c1"
format_stamp: "Formatted at 2026-09-27 18:50:47 on dist-test-slave-8tg2"
server_key: "4c59d28d5bcf728396ef2fab1b17938f"
server_key_iv: "1965b538c480be5ba9117a3540437d39"
server_key_version: "encryptionkey@0"
I20260927 18:50:47.733872 16552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 18:50:47.736213 16566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:47.737430 16552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:50:47.737560 16552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "58d5656ab80340558b106131c9bcf7c1"
format_stamp: "Formatted at 2026-09-27 18:50:47 on dist-test-slave-8tg2"
server_key: "4c59d28d5bcf728396ef2fab1b17938f"
server_key_iv: "1965b538c480be5ba9117a3540437d39"
server_key_version: "encryptionkey@0"
I20260927 18:50:47.737668 16552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:47.751461 16552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:47.752079 16552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:47.752262 16552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:47.752801 16552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:47.753738 16552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:47.753805 16552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:47.753872 16552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:47.753926 16552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:47.762507 16552 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:43789
I20260927 18:50:47.762544 16679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:43789 every 8 connection(s)
I20260927 18:50:47.763342 16552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:50:47.763908 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 16552
I20260927 18:50:47.764041 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:50:47.764355 30333 external_mini_cluster.cc:1428] Setting key 6673f8a771e558a9bcc50581313db9a5
I20260927 18:50:47.766444 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34005
--builtin_ntp_servers=127.29.159.84:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:50:47.773300 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.765321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:59299 (local address 127.29.159.126:34005)
0927 18:50:47.765452 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:47.765456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:47.766700 (+ 1244us) server_negotiation.cc:407] Connection header received
0927 18:50:47.768442 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:47.768445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:47.768494 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:47.768573 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:47.769514 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.770052 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.771215 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.771520 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.771635 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.772782 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:47.772785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:47.772787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:47.772799 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:47.772826 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:47.772834 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:47.772910 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:47.773094 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 18:50:47.773156 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:50:47.774140 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.765810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:34005 (local address 127.29.159.65:59299)
0927 18:50:47.766555 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:47.766608 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:47.768123 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:47.768789 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:47.768800 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:47.768912 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:47.769385 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.769406 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.770195 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.770199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.771080 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.771090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.771675 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.771679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.771845 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.772508 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:50:47.772509 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:47.772549 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:50:47.772612 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:47.772952 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:47.772960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:47.772964 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:50:47.773091 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 18:50:47.773370 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":130,"threads_started":1}
I20260927 18:50:47.775552 16680 heartbeater.cc:347] Connected to a master server at 127.29.159.126:34005
I20260927 18:50:47.775789 16680 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:47.776298 16680 heartbeater.cc:511] Master 127.29.159.126:34005 requested a full tablet report, sending...
I20260927 18:50:47.778020 16492 ts_manager.cc:194] Registered new tserver with Master: 58d5656ab80340558b106131c9bcf7c1 (127.29.159.65:43789)
I20260927 18:50:47.779613 16492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.65:59299
W20260927 18:50:47.878610 16683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:47.878887 16683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:47.878929 16683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:47.882992 16683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:47.883047 16683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:47.883104 16683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:50:47.887851 16683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:47.888911 16683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:47.890276 16683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:47.900008 16691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:47.900061 16688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:47.900020 16689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:47.900240 16683 server_base.cc:1034] running on GCE node
I20260927 18:50:47.900794 16683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:47.901473 16683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:47.902662 16683 hybrid_clock.cc:648] HybridClock initialized: now 1790535047902643 us; error 30 us; skew 500 ppm
I20260927 18:50:47.904596 16683 webserver.cc:460] Webserver started at http://127.29.159.66:42871/ using document root <none> and password file <none>
I20260927 18:50:47.905220 16683 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:47.905313 16683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:47.905544 16683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:47.907660 16683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "f6b9a57fa44a46a5af49966ee4059a98"
format_stamp: "Formatted at 2026-09-27 18:50:47 on dist-test-slave-8tg2"
server_key: "623c2545a707183ab28697a3ac416f63"
server_key_iv: "9b64ba526adfe7370f52d6915072aeae"
server_key_version: "encryptionkey@0"
I20260927 18:50:47.908259 16683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "f6b9a57fa44a46a5af49966ee4059a98"
format_stamp: "Formatted at 2026-09-27 18:50:47 on dist-test-slave-8tg2"
server_key: "623c2545a707183ab28697a3ac416f63"
server_key_iv: "9b64ba526adfe7370f52d6915072aeae"
server_key_version: "encryptionkey@0"
I20260927 18:50:47.911837 16683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:50:47.914357 16697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:47.915659 16683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:47.915786 16683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "f6b9a57fa44a46a5af49966ee4059a98"
format_stamp: "Formatted at 2026-09-27 18:50:47 on dist-test-slave-8tg2"
server_key: "623c2545a707183ab28697a3ac416f63"
server_key_iv: "9b64ba526adfe7370f52d6915072aeae"
server_key_version: "encryptionkey@0"
I20260927 18:50:47.915899 16683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:47.933313 16683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:47.933997 16683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:47.934226 16683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:47.934819 16683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:47.935801 16683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:47.935868 16683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:47.935962 16683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:47.936013 16683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:47.944844 16683 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:45071
I20260927 18:50:47.944872 16810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:45071 every 8 connection(s)
I20260927 18:50:47.945698 16683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:50:47.953856 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 16683
I20260927 18:50:47.954010 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:50:47.954350 30333 external_mini_cluster.cc:1428] Setting key 48160f6f8d2d321098acbd89866b4549
I20260927 18:50:47.955106 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.947419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:46951 (local address 127.29.159.126:34005)
0927 18:50:47.947569 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:47.947573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:47.948612 (+ 1039us) server_negotiation.cc:407] Connection header received
0927 18:50:47.950054 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:47.950056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:47.950104 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:47.950179 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:47.951197 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.951836 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.953014 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.953217 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.953289 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.954581 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:47.954585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:47.954587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:47.954599 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:47.954615 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:47.954622 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:47.954681 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:47.954891 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 18:50:47.954952 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 18:50:47.956110 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:47.947840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:34005 (local address 127.29.159.66:46951)
0927 18:50:47.948469 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:47.948508 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:47.949858 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:47.950378 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:47.950406 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:47.950521 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:47.951076 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.951089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.951960 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.951962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.952910 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:47.952920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:47.953392 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:47.953395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:47.953533 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:47.954211 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:50:47.954214 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:47.954282 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:50:47.954372 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:47.954711 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:47.954719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:47.954725 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:50:47.954887 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 18:50:47.955163 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
I20260927 18:50:47.956542 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34005
--builtin_ntp_servers=127.29.159.84:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:50:47.957285 16811 heartbeater.cc:347] Connected to a master server at 127.29.159.126:34005
I20260927 18:50:47.957527 16811 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:47.958132 16811 heartbeater.cc:511] Master 127.29.159.126:34005 requested a full tablet report, sending...
I20260927 18:50:47.959223 16492 ts_manager.cc:194] Registered new tserver with Master: f6b9a57fa44a46a5af49966ee4059a98 (127.29.159.66:45071)
I20260927 18:50:47.960080 16492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.66:46951
W20260927 18:50:48.070591 16814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:48.070861 16814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:48.070904 16814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:48.074896 16814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:48.074952 16814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:48.075011 16814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:50:48.080018 16814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:48.081059 16814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:48.082368 16814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:48.091953 16822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:48.092034 16820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:48.091948 16819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:48.092242 16814 server_base.cc:1034] running on GCE node
I20260927 18:50:48.092631 16814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:48.093321 16814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:48.094537 16814 hybrid_clock.cc:648] HybridClock initialized: now 1790535048094500 us; error 48 us; skew 500 ppm
I20260927 18:50:48.096692 16814 webserver.cc:460] Webserver started at http://127.29.159.67:44987/ using document root <none> and password file <none>
I20260927 18:50:48.097390 16814 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:48.097489 16814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:48.097728 16814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:48.099957 16814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "c2bdd1f4375343a38b1ef6004f50b184"
format_stamp: "Formatted at 2026-09-27 18:50:48 on dist-test-slave-8tg2"
server_key: "9d762e36beafc2f1334cdc08e4c72ad1"
server_key_iv: "86ae810535dc8b9aa6c992bab3eec436"
server_key_version: "encryptionkey@0"
I20260927 18:50:48.100555 16814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "c2bdd1f4375343a38b1ef6004f50b184"
format_stamp: "Formatted at 2026-09-27 18:50:48 on dist-test-slave-8tg2"
server_key: "9d762e36beafc2f1334cdc08e4c72ad1"
server_key_iv: "86ae810535dc8b9aa6c992bab3eec436"
server_key_version: "encryptionkey@0"
I20260927 18:50:48.104321 16814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:50:48.106948 16828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:48.108290 16814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:48.108496 16814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "c2bdd1f4375343a38b1ef6004f50b184"
format_stamp: "Formatted at 2026-09-27 18:50:48 on dist-test-slave-8tg2"
server_key: "9d762e36beafc2f1334cdc08e4c72ad1"
server_key_iv: "86ae810535dc8b9aa6c992bab3eec436"
server_key_version: "encryptionkey@0"
I20260927 18:50:48.108623 16814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:48.122053 16814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:48.122673 16814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:48.122820 16814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:48.123337 16814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:48.124301 16814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:48.124356 16814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:48.124396 16814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:48.124428 16814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:48.133020 16814 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:41233
I20260927 18:50:48.133116 16941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:41233 every 8 connection(s)
I20260927 18:50:48.133989 16814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:50:48.143018 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.135869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:45141 (local address 127.29.159.126:34005)
0927 18:50:48.136018 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:48.136022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:48.137106 (+ 1084us) server_negotiation.cc:407] Connection header received
0927 18:50:48.138475 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:48.138478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:48.138524 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:48.138601 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:48.139407 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.139992 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.141188 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.141404 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.141506 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.142614 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:48.142617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:48.142619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:48.142630 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:48.142647 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:48.142654 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:48.142713 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:48.142849 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 18:50:48.142896 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:50:48.143698 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 16814
I20260927 18:50:48.143810 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:50:48.144013 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.136322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:34005 (local address 127.29.159.67:45141)
0927 18:50:48.136963 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:48.137003 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:48.138274 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:48.138733 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:48.138740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:48.138823 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:48.139292 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.139303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.140108 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.140110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.141087 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.141097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.141590 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.141593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.141727 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.142373 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:50:48.142374 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:48.142413 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:50:48.142485 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:48.142729 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:48.142735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:48.142739 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:50:48.142837 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:50:48.143047 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
I20260927 18:50:48.144132 30333 external_mini_cluster.cc:1428] Setting key b75c041c9485e8db1966f622ceed00fb
I20260927 18:50:48.145187 16942 heartbeater.cc:347] Connected to a master server at 127.29.159.126:34005
I20260927 18:50:48.145437 16942 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:48.146019 16942 heartbeater.cc:511] Master 127.29.159.126:34005 requested a full tablet report, sending...
I20260927 18:50:48.147027 16491 ts_manager.cc:194] Registered new tserver with Master: c2bdd1f4375343a38b1ef6004f50b184 (127.29.159.67:41233)
I20260927 18:50:48.147639 16491 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.67:45141
I20260927 18:50:48.158221 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:50:48.164028 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.160299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.29.159.126:34005)
0927 18:50:48.160450 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:48.160453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:48.160554 (+ 101us) server_negotiation.cc:407] Connection header received
0927 18:50:48.160649 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:48.160651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:48.160689 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:48.160763 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:48.161049 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.161599 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.162461 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.162672 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.162752 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.163596 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:48.163599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:48.163601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:48.163612 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:48.163631 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:48.163639 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:48.163746 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:48.163875 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 18:50:48.163910 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:50:48.167151 16491 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:50:48.170125 16491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:50:48.184381 16956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.178807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.29.159.65:43789)
0927 18:50:48.179218 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:48.179230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:48.179997 (+ 767us) server_negotiation.cc:407] Connection header received
0927 18:50:48.180070 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:48.180074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:48.180260 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:48.180409 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:48.180847 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.181570 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.183008 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.183653 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.183763 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.183963 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:48.184132 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 18:50:48.184231 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":131,"threads_started":1}
I20260927 18:50:48.184376 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.178567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:43789 (local address 127.0.0.1:38180)
0927 18:50:48.179848 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:48.179861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:48.179938 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:48.180544 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:48.180546 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:48.180561 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:48.180733 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.180737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.181688 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.181693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.182886 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.182895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.183776 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.183780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.183942 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.183990 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 18:50:48.184098 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 18:50:48.184216 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":93,"thread_start_us":69,"threads_started":1}
I20260927 18:50:48.185115 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.179220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:45071 (local address 127.0.0.1:56118)
0927 18:50:48.179593 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:48.179609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:48.179704 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:48.180377 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:48.180383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:48.180404 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:48.180550 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.180557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.182149 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.182152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.183367 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.183378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.184406 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.184409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.184848 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.184875 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:50:48.184928 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 18:50:48.184994 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":38,"thread_start_us":86,"threads_started":1}
I20260927 18:50:48.185433 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.178567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:41233 (local address 127.0.0.1:49378)
0927 18:50:48.179467 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:48.179523 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:48.179645 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:48.180565 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:48.180568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:48.180583 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:48.180777 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.180783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.182148 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.182151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.183948 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.183956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.185012 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.185016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.185099 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.185121 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:50:48.185198 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:50:48.185263 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":262,"thread_start_us":93,"threads_started":1}
I20260927 18:50:48.185787 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.179512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.29.159.66:45071)
0927 18:50:48.179948 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:48.179964 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:48.179989 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:50:48.180069 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:48.180073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:48.180226 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:48.180387 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:48.180687 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.182034 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.183517 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.184260 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.185322 (+ 1062us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.185502 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:48.185550 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 18:50:48.185638 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":131,"threads_started":1}
I20260927 18:50:48.187636 16614 tablet_service.cc:1467] Processing CreateTablet for tablet d177d3b0a15240978c1ff932a5f6bb1b (DEFAULT_TABLE table=test-table [id=8f7bd798d80f425c982eb5d52c1d5983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:48.188794 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d177d3b0a15240978c1ff932a5f6bb1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:48.188431 16745 tablet_service.cc:1467] Processing CreateTablet for tablet d177d3b0a15240978c1ff932a5f6bb1b (DEFAULT_TABLE table=test-table [id=8f7bd798d80f425c982eb5d52c1d5983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:48.189420 16745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d177d3b0a15240978c1ff932a5f6bb1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:48.192076 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.178732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.29.159.67:41233)
0927 18:50:48.179934 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:48.179950 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:48.179981 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:50:48.180060 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:48.180065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:48.180260 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:48.180409 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:48.180903 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.182016 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.184057 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.184848 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.184956 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.185126 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:48.191792 (+ 6666us) server_negotiation.cc:299] Negotiation successful
0927 18:50:48.191903 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1094,"thread_start_us":139,"threads_started":1}
I20260927 18:50:48.194509 16876 tablet_service.cc:1467] Processing CreateTablet for tablet d177d3b0a15240978c1ff932a5f6bb1b (DEFAULT_TABLE table=test-table [id=8f7bd798d80f425c982eb5d52c1d5983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:48.195343 16876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d177d3b0a15240978c1ff932a5f6bb1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:48.196789 16960 tablet_bootstrap.cc:492] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1: Bootstrap starting.
I20260927 18:50:48.197070 16959 tablet_bootstrap.cc:492] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98: Bootstrap starting.
I20260927 18:50:48.198873 16959 tablet_bootstrap.cc:654] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:48.199016 16960 tablet_bootstrap.cc:654] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:48.199702 16959 log.cc:824] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:48.199851 16960 log.cc:824] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:48.201709 16959 tablet_bootstrap.cc:492] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98: No bootstrap required, opened a new log
I20260927 18:50:48.201915 16959 ts_tablet_manager.cc:1397] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 18:50:48.203145 16963 tablet_bootstrap.cc:492] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184: Bootstrap starting.
I20260927 18:50:48.204787 16960 tablet_bootstrap.cc:492] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1: No bootstrap required, opened a new log
I20260927 18:50:48.204998 16960 ts_tablet_manager.cc:1397] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 18:50:48.205372 16963 tablet_bootstrap.cc:654] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:48.206125 16959 raft_consensus.cc:348] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2bdd1f4375343a38b1ef6004f50b184" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41233 } } peers { permanent_uuid: "58d5656ab80340558b106131c9bcf7c1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 43789 } } peers { permanent_uuid: "f6b9a57fa44a46a5af49966ee4059a98" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45071 } }
I20260927 18:50:48.206403 16959 raft_consensus.cc:374] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:48.206400 16963 log.cc:824] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:48.206456 16959 raft_consensus.cc:729] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b9a57fa44a46a5af49966ee4059a98, State: Initialized, Role: FOLLOWER
I20260927 18:50:48.206959 16959 consensus_queue.cc:260] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2bdd1f4375343a38b1ef6004f50b184" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41233 } } peers { permanent_uuid: "58d5656ab80340558b106131c9bcf7c1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 43789 } } peers { permanent_uuid: "f6b9a57fa44a46a5af49966ee4059a98" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45071 } }
I20260927 18:50:48.207736 16811 heartbeater.cc:503] Master 127.29.159.126:34005 was elected leader, sending a full tablet report...
I20260927 18:50:48.207994 16959 ts_tablet_manager.cc:1428] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 18:50:48.208345 16963 tablet_bootstrap.cc:492] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184: No bootstrap required, opened a new log
I20260927 18:50:48.208474 16963 ts_tablet_manager.cc:1397] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:50:48.209025 16960 raft_consensus.cc:348] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2bdd1f4375343a38b1ef6004f50b184" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41233 } } peers { permanent_uuid: "58d5656ab80340558b106131c9bcf7c1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 43789 } } peers { permanent_uuid: "f6b9a57fa44a46a5af49966ee4059a98" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45071 } }
I20260927 18:50:48.209255 16960 raft_consensus.cc:374] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:48.209331 16960 raft_consensus.cc:729] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d5656ab80340558b106131c9bcf7c1, State: Initialized, Role: FOLLOWER
I20260927 18:50:48.209775 16960 consensus_queue.cc:260] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2bdd1f4375343a38b1ef6004f50b184" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41233 } } peers { permanent_uuid: "58d5656ab80340558b106131c9bcf7c1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 43789 } } peers { permanent_uuid: "f6b9a57fa44a46a5af49966ee4059a98" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45071 } }
I20260927 18:50:48.210405 16680 heartbeater.cc:503] Master 127.29.159.126:34005 was elected leader, sending a full tablet report...
I20260927 18:50:48.210762 16960 ts_tablet_manager.cc:1428] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:50:48.212978 16963 raft_consensus.cc:348] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2bdd1f4375343a38b1ef6004f50b184" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41233 } } peers { permanent_uuid: "58d5656ab80340558b106131c9bcf7c1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 43789 } } peers { permanent_uuid: "f6b9a57fa44a46a5af49966ee4059a98" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45071 } }
I20260927 18:50:48.213207 16963 raft_consensus.cc:374] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:48.213267 16963 raft_consensus.cc:729] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2bdd1f4375343a38b1ef6004f50b184, State: Initialized, Role: FOLLOWER
I20260927 18:50:48.213716 16963 consensus_queue.cc:260] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2bdd1f4375343a38b1ef6004f50b184" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41233 } } peers { permanent_uuid: "58d5656ab80340558b106131c9bcf7c1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 43789 } } peers { permanent_uuid: "f6b9a57fa44a46a5af49966ee4059a98" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45071 } }
I20260927 18:50:48.214579 16963 ts_tablet_manager.cc:1428] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:50:48.214871 16942 heartbeater.cc:503] Master 127.29.159.126:34005 was elected leader, sending a full tablet report...
I20260927 18:50:48.230019 16967 raft_consensus.cc:482] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:48.230201 16967 raft_consensus.cc:504] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2bdd1f4375343a38b1ef6004f50b184" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41233 } } peers { permanent_uuid: "58d5656ab80340558b106131c9bcf7c1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 43789 } } peers { permanent_uuid: "f6b9a57fa44a46a5af49966ee4059a98" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45071 } }
I20260927 18:50:48.231302 16967 leader_election.cc:288] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58d5656ab80340558b106131c9bcf7c1 (127.29.159.65:43789), f6b9a57fa44a46a5af49966ee4059a98 (127.29.159.66:45071)
I20260927 18:50:48.236114 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.231623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:43789 (local address 127.29.159.67:55397)
0927 18:50:48.231809 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:48.231836 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:48.232002 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:48.232421 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:48.232424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:48.232445 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:48.232564 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.232571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.233608 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.233611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.234942 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.234953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.235708 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.235712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.235823 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.235860 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:50:48.235939 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:50:48.236010 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 18:50:48.236800 16956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.231742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:55397 (local address 127.29.159.65:43789)
0927 18:50:48.231908 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:48.231913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:48.231933 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:50:48.232149 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:48.232153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:48.232210 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:48.232281 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:48.232702 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.233476 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.235129 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.235597 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.235731 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.235827 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:48.236648 (+ 821us) server_negotiation.cc:299] Negotiation successful
0927 18:50:48.236710 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:50:48.237001 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.231627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:45071 (local address 127.29.159.67:46007)
0927 18:50:48.232824 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:48.232870 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:48.233005 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:48.233499 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:48.233502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:48.233522 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:48.233640 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.233647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.234782 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.234787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.235783 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:48.235791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.236601 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.236604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:48.236672 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.236690 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 18:50:48.236779 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:50:48.236845 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":566,"thread_start_us":158,"threads_started":1}
I20260927 18:50:48.237007 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.231741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:46007 (local address 127.29.159.66:45071)
0927 18:50:48.232171 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:48.232178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:48.232974 (+ 796us) server_negotiation.cc:407] Connection header received
0927 18:50:48.233236 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:48.233240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:48.233296 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:48.233359 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:48.233776 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.234623 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.235952 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.236474 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.236571 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.236604 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:48.236798 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 18:50:48.236896 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352}
I20260927 18:50:48.237397 16634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d177d3b0a15240978c1ff932a5f6bb1b" candidate_uuid: "c2bdd1f4375343a38b1ef6004f50b184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d5656ab80340558b106131c9bcf7c1" is_pre_election: true
I20260927 18:50:48.237725 16634 raft_consensus.cc:2454] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2bdd1f4375343a38b1ef6004f50b184 in term 0.
I20260927 18:50:48.237717 16765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d177d3b0a15240978c1ff932a5f6bb1b" candidate_uuid: "c2bdd1f4375343a38b1ef6004f50b184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b9a57fa44a46a5af49966ee4059a98" is_pre_election: true
I20260927 18:50:48.237995 16765 raft_consensus.cc:2454] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2bdd1f4375343a38b1ef6004f50b184 in term 0.
I20260927 18:50:48.238226 16832 leader_election.cc:302] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d5656ab80340558b106131c9bcf7c1, c2bdd1f4375343a38b1ef6004f50b184; no voters:
I20260927 18:50:48.238561 16967 raft_consensus.cc:2781] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:50:48.238652 16967 raft_consensus.cc:482] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:50:48.238685 16967 raft_consensus.cc:3037] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:48.239661 16967 raft_consensus.cc:504] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2bdd1f4375343a38b1ef6004f50b184" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41233 } } peers { permanent_uuid: "58d5656ab80340558b106131c9bcf7c1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 43789 } } peers { permanent_uuid: "f6b9a57fa44a46a5af49966ee4059a98" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45071 } }
I20260927 18:50:48.239974 16967 leader_election.cc:288] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [CANDIDATE]: Term 1 election: Requested vote from peers 58d5656ab80340558b106131c9bcf7c1 (127.29.159.65:43789), f6b9a57fa44a46a5af49966ee4059a98 (127.29.159.66:45071)
I20260927 18:50:48.240270 16634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d177d3b0a15240978c1ff932a5f6bb1b" candidate_uuid: "c2bdd1f4375343a38b1ef6004f50b184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d5656ab80340558b106131c9bcf7c1"
I20260927 18:50:48.240319 16765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d177d3b0a15240978c1ff932a5f6bb1b" candidate_uuid: "c2bdd1f4375343a38b1ef6004f50b184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b9a57fa44a46a5af49966ee4059a98"
I20260927 18:50:48.240379 16634 raft_consensus.cc:3037] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:48.240468 16765 raft_consensus.cc:3037] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:48.241518 16634 raft_consensus.cc:2454] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2bdd1f4375343a38b1ef6004f50b184 in term 1.
I20260927 18:50:48.241752 16765 raft_consensus.cc:2454] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2bdd1f4375343a38b1ef6004f50b184 in term 1.
I20260927 18:50:48.241875 16832 leader_election.cc:302] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d5656ab80340558b106131c9bcf7c1, c2bdd1f4375343a38b1ef6004f50b184; no voters:
I20260927 18:50:48.242048 16967 raft_consensus.cc:2781] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:48.242302 16967 raft_consensus.cc:686] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 1 LEADER]: Becoming Leader. State: Replica: c2bdd1f4375343a38b1ef6004f50b184, State: Running, Role: LEADER
I20260927 18:50:48.242648 16967 consensus_queue.cc:237] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2bdd1f4375343a38b1ef6004f50b184" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41233 } } peers { permanent_uuid: "58d5656ab80340558b106131c9bcf7c1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 43789 } } peers { permanent_uuid: "f6b9a57fa44a46a5af49966ee4059a98" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45071 } }
I20260927 18:50:48.246097 16491 catalog_manager.cc:5495] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2bdd1f4375343a38b1ef6004f50b184 (127.29.159.67). New cstate: current_term: 1 leader_uuid: "c2bdd1f4375343a38b1ef6004f50b184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2bdd1f4375343a38b1ef6004f50b184" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 41233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d5656ab80340558b106131c9bcf7c1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 43789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6b9a57fa44a46a5af49966ee4059a98" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45071 } health_report { overall_health: UNKNOWN } } }
W20260927 18:50:48.264602 16681 tablet.cc:2369] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:50:48.270598 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.266888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.29.159.67:41233)
0927 18:50:48.267037 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:48.267041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:48.267058 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:50:48.267129 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:48.267131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:48.267184 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:48.267272 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:48.267683 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.268175 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.269045 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:48.269255 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:48.269332 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:48.269612 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:48.270024 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:48.270134 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:48.270382 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 18:50:48.270413 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:50:48.276664 16634 raft_consensus.cc:1261] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1 [term 1 FOLLOWER]: Refusing update from remote peer c2bdd1f4375343a38b1ef6004f50b184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:48.276664 16765 raft_consensus.cc:1261] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98 [term 1 FOLLOWER]: Refusing update from remote peer c2bdd1f4375343a38b1ef6004f50b184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:48.277456 16969 consensus_queue.cc:1035] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d5656ab80340558b106131c9bcf7c1" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:48.277673 16967 consensus_queue.cc:1035] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b9a57fa44a46a5af49966ee4059a98" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:48.295329 16491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:48.295545 16491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:48.298084 16491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bcad2e56c4e4edfa958d71a0ae6b974: Sending DeleteTablet for 3 replicas of tablet d177d3b0a15240978c1ff932a5f6bb1b
I20260927 18:50:48.298844 16876 tablet_service.cc:1514] Processing DeleteTablet for tablet d177d3b0a15240978c1ff932a5f6bb1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:48 UTC) from {username='slave'} at 127.0.0.1:49378
I20260927 18:50:48.298861 16614 tablet_service.cc:1514] Processing DeleteTablet for tablet d177d3b0a15240978c1ff932a5f6bb1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:48 UTC) from {username='slave'} at 127.0.0.1:38180
I20260927 18:50:48.299005 16745 tablet_service.cc:1514] Processing DeleteTablet for tablet d177d3b0a15240978c1ff932a5f6bb1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:48 UTC) from {username='slave'} at 127.0.0.1:56118
I20260927 18:50:48.299413 16983 tablet_replica.cc:330] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1: stopping tablet replica
I20260927 18:50:48.299633 16984 tablet_replica.cc:330] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98: stopping tablet replica
I20260927 18:50:48.299820 16982 tablet_replica.cc:330] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184: stopping tablet replica
I20260927 18:50:48.299948 16983 raft_consensus.cc:2229] T d177d3b0a15240978c1ff932a5f6bb1b P 58d5656ab80340558b106131c9bcf7c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:48.300096 16982 raft_consensus.cc:2229] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:50:48.300215 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 16552
I20260927 18:50:48.300863 16982 raft_consensus.cc:2258] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:48.302372 16982 ts_tablet_manager.cc:1905] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:48.304234 16984 raft_consensus.cc:2229] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:48.304616 16984 raft_consensus.cc:2258] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:48.305213 16982 ts_tablet_manager.cc:1918] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:48.305305 16982 log.cc:1196] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/d177d3b0a15240978c1ff932a5f6bb1b
I20260927 18:50:48.305627 16982 ts_tablet_manager.cc:1939] T d177d3b0a15240978c1ff932a5f6bb1b P c2bdd1f4375343a38b1ef6004f50b184: Deleting consensus metadata
I20260927 18:50:48.306459 16984 ts_tablet_manager.cc:1905] T d177d3b0a15240978c1ff932a5f6bb1b P f6b9a57fa44a46a5af49966ee4059a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:48.306783 16477 catalog_manager.cc:4841] TS c2bdd1f4375343a38b1ef6004f50b184 (127.29.159.67:41233): tablet d177d3b0a15240978c1ff932a5f6bb1b (table test-table [id=8f7bd798d80f425c982eb5d52c1d5983]) successfully deleted
W20260927 18:50:48.308254 16479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:43789 (error 108)
I20260927 18:50:48.308475 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 16683
I20260927 18:50:48.309497 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.309154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:43789 (local address 127.0.0.1:38196)
0927 18:50:48.309314 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:48.309394 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:43789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 18:50:48.309666 16479 catalog_manager.cc:4584] TS 58d5656ab80340558b106131c9bcf7c1 (127.29.159.65:43789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d177d3b0a15240978c1ff932a5f6bb1b: Network error: Client connection negotiation failed: client connection to 127.29.159.65:43789: connect: Connection refused (error 111)
I20260927 18:50:48.316076 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 16814
I20260927 18:50:48.316287 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:48.315978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:45071 (local address 127.0.0.1:56134)
0927 18:50:48.316121 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:48.316191 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.66:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 18:50:48.321933 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 16461
2026-09-27T18:50:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1014 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:48 dist-test-slave-8tg2 krb5kdc[16988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:48 dist-test-slave-8tg2 krb5kdc[16988](info): set up 2 sockets
Sep 27 18:50:48 dist-test-slave-8tg2 krb5kdc[16988](info): commencing operation
krb5kdc: starting...
W20260927 18:50:49.363888 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:50:49 dist-test-slave-8tg2 krb5kdc[16988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:50:49.490149 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:33671
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:33671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:49.600380 17001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:49.600660 17001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:49.600703 17001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:49.604707 17001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:49.604760 17001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:49.604780 17001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:49.604797 17001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:49.604813 17001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:49.610073 17001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:33671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:33671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:49.611106 17001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:49.612121 17001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:49.620733 17006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:49.620759 17009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:49.620848 17007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:49.621023 17001 server_base.cc:1034] running on GCE node
I20260927 18:50:49.621598 17001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:49.622691 17001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:49.623847 17001 hybrid_clock.cc:648] HybridClock initialized: now 1790535049623831 us; error 29 us; skew 500 ppm
Sep 27 18:50:49 dist-test-slave-8tg2 krb5kdc[16988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535049, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:49.626504 17001 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:50:49.627349 17001 webserver.cc:460] Webserver started at http://127.29.159.126:35075/ using document root <none> and password file <none>
I20260927 18:50:49.627938 17001 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:49.628028 17001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:49.628258 17001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:49.630337 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "ae264e60120d46bb8e90d05c1351df70"
format_stamp: "Formatted at 2026-09-27 18:50:49 on dist-test-slave-8tg2"
server_key: "ae10c89700999edd13808ef96bcd591e"
server_key_iv: "13e4b936a6873a12c604c4fc2b92f323"
server_key_version: "encryptionkey@0"
I20260927 18:50:49.630919 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "ae264e60120d46bb8e90d05c1351df70"
format_stamp: "Formatted at 2026-09-27 18:50:49 on dist-test-slave-8tg2"
server_key: "ae10c89700999edd13808ef96bcd591e"
server_key_iv: "13e4b936a6873a12c604c4fc2b92f323"
server_key_version: "encryptionkey@0"
I20260927 18:50:49.634585 17001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 18:50:49.637087 17016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:49.638353 17001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:50:49.638504 17001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "ae264e60120d46bb8e90d05c1351df70"
format_stamp: "Formatted at 2026-09-27 18:50:49 on dist-test-slave-8tg2"
server_key: "ae10c89700999edd13808ef96bcd591e"
server_key_iv: "13e4b936a6873a12c604c4fc2b92f323"
server_key_version: "encryptionkey@0"
I20260927 18:50:49.638618 17001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:49.656049 17001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:49.659056 17001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:49.659279 17001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:49.666627 17001 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:33671
I20260927 18:50:49.666656 17068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:33671 every 8 connection(s)
I20260927 18:50:49.667626 17001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:49.670604 17069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:49.676285 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 17001
I20260927 18:50:49.676440 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:50:49.676652 17069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70: Bootstrap starting.
I20260927 18:50:49.676779 30333 external_mini_cluster.cc:1428] Setting key 843ae2bd2ab3b4f739aaa4d341e77334
I20260927 18:50:49.679342 17069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:49.680161 17069 log.cc:824] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:49.682117 17069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70: No bootstrap required, opened a new log
Sep 27 18:50:49 dist-test-slave-8tg2 krb5kdc[16988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:49.686576 17069 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae264e60120d46bb8e90d05c1351df70" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33671 } }
I20260927 18:50:49.686857 17069 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:49.686933 17069 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae264e60120d46bb8e90d05c1351df70, State: Initialized, Role: FOLLOWER
I20260927 18:50:49.687526 17069 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae264e60120d46bb8e90d05c1351df70" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33671 } }
I20260927 18:50:49.687691 17069 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:49.687790 17069 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:49.687893 17069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:49.688827 17069 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae264e60120d46bb8e90d05c1351df70" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33671 } }
I20260927 18:50:49.689174 17069 leader_election.cc:302] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae264e60120d46bb8e90d05c1351df70; no voters:
I20260927 18:50:49.689493 17069 leader_election.cc:288] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:49.689615 17074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:49.689834 17074 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [term 1 LEADER]: Becoming Leader. State: Replica: ae264e60120d46bb8e90d05c1351df70, State: Running, Role: LEADER
I20260927 18:50:49.690165 17074 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae264e60120d46bb8e90d05c1351df70" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33671 } }
I20260927 18:50:49.690949 17069 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:49.691761 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:49.678243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.29.159.126:33671)
0927 18:50:49.678696 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:49.678711 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:49.678747 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:50:49.679460 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:49.679496 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:49.679853 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:49.680201 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:49.681022 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:49.682016 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:49.682939 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:49.683412 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:49.683674 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:49.686431 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:49.686437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:49.686451 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:49.686499 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:49.688438 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:49.688906 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:49.688915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:49.688922 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:49.689047 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:49.689325 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:49.689329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:49.689330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:49.689939 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:49.690149 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:49.690492 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 18:50:49.690795 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":123,"threads_started":1}
I20260927 18:50:49.691900 17075 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae264e60120d46bb8e90d05c1351df70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae264e60120d46bb8e90d05c1351df70" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33671 } } }
I20260927 18:50:49.692046 17075 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:49.692221 17076 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae264e60120d46bb8e90d05c1351df70. Latest consensus state: current_term: 1 leader_uuid: "ae264e60120d46bb8e90d05c1351df70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae264e60120d46bb8e90d05c1351df70" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 33671 } } }
I20260927 18:50:49.692312 17076 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:49.692564 17083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:49.696295 17083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:49.702178 17083 catalog_manager.cc:1347] Generated new cluster ID: b6c7b268f6cc4aa1b9036b74b59ebbbc
I20260927 18:50:49.702277 17083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:49.715929 17083 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:49.717000 17083 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:49.724326 17083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70: Generated new TSK 0
I20260927 18:50:49.725191 17083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.159.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
Entry for principal kudu/127.29.159.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab.
I20260927 18:50:49.762651 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33671
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:49.875895 17095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:49.876217 17095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:49.876294 17095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:49.880473 17095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:49.880560 17095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:49.880656 17095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:50:49.886025 17095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.65.keytab
--principal=kudu/127.29.159.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:49.887173 17095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:49.888334 17095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:49.898918 17103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:49.898918 17100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:49.898988 17095 server_base.cc:1034] running on GCE node
W20260927 18:50:49.899211 17101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:49.899716 17095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:49.900416 17095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:49.901597 17095 hybrid_clock.cc:648] HybridClock initialized: now 1790535049901574 us; error 38 us; skew 500 ppm
Sep 27 18:50:49 dist-test-slave-8tg2 krb5kdc[16988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535049, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:49.904724 17095 init.cc:377] Logged in from keytab as kudu/127.29.159.65@KRBTEST.COM (short username kudu)
I20260927 18:50:49.905707 17095 webserver.cc:460] Webserver started at http://127.29.159.65:40969/ using document root <none> and password file <none>
I20260927 18:50:49.906322 17095 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:49.906389 17095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:49.906657 17095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:49.908905 17095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "aa80f6e496294bee81069297e91026f9"
format_stamp: "Formatted at 2026-09-27 18:50:49 on dist-test-slave-8tg2"
server_key: "15a301b523f7d1b7fa84c4cb8fe061dd"
server_key_iv: "539f9c63b117a0939cea93016512affd"
server_key_version: "encryptionkey@0"
I20260927 18:50:49.909497 17095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "aa80f6e496294bee81069297e91026f9"
format_stamp: "Formatted at 2026-09-27 18:50:49 on dist-test-slave-8tg2"
server_key: "15a301b523f7d1b7fa84c4cb8fe061dd"
server_key_iv: "539f9c63b117a0939cea93016512affd"
server_key_version: "encryptionkey@0"
I20260927 18:50:49.913385 17095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:50:49.915989 17110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:49.917354 17095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:50:49.917522 17095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "aa80f6e496294bee81069297e91026f9"
format_stamp: "Formatted at 2026-09-27 18:50:49 on dist-test-slave-8tg2"
server_key: "15a301b523f7d1b7fa84c4cb8fe061dd"
server_key_iv: "539f9c63b117a0939cea93016512affd"
server_key_version: "encryptionkey@0"
I20260927 18:50:49.917654 17095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:49.933848 17095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:49.937258 17095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:49.937453 17095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:49.938090 17095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:49.938910 17095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:49.938956 17095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:49.939002 17095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:49.939018 17095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:49.948526 17095 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:38621
I20260927 18:50:49.948544 17223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:38621 every 8 connection(s)
I20260927 18:50:49.949455 17095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:50:49.949759 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 17095
I20260927 18:50:49.949911 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:50:49.950273 30333 external_mini_cluster.cc:1428] Setting key 3f892b9f09ddfb9dd0aeeee1a5ca4bf7
WARNING: no policy specified for kudu/127.29.159.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.66@KRBTEST.COM" created.
Sep 27 18:50:49 dist-test-slave-8tg2 krb5kdc[16988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535049, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.65@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:49.968801 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:49.951623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:41237 (local address 127.29.159.126:33671)
0927 18:50:49.951777 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:49.951781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:49.952927 (+ 1146us) server_negotiation.cc:407] Connection header received
0927 18:50:49.954597 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:49.954602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:49.954662 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:49.954766 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:49.956041 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:49.956965 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:49.958970 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:49.959208 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:49.959292 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:49.964993 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:49.964997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:49.965000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:49.965029 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:49.966568 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:49.967128 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:49.967138 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:49.967140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:49.967218 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:49.967644 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:49.967647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:49.967649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:49.967840 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:49.967947 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:49.968493 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 18:50:49.968625 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:50:49.969632 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:49.952182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:33671 (local address 127.29.159.65:41237)
0927 18:50:49.952711 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:49.952773 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:49.954383 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:49.954940 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:49.954947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:49.955303 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:49.955885 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:49.955900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:49.957152 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:49.957158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:49.958802 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:49.958845 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:49.959381 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:49.959389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:49.959652 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:49.960364 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:49.960390 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:49.964749 (+ 4359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:49.966703 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:49.966709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:49.966727 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:49.967005 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:49.967311 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:49.967316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:49.967318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:49.967534 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:49.967958 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:49.967966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:49.968239 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 18:50:49.968431 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 18:50:49.968788 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":100,"threads_started":1}
I20260927 18:50:49.970950 17224 heartbeater.cc:347] Connected to a master server at 127.29.159.126:33671
I20260927 18:50:49.971254 17224 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:49.971982 17224 heartbeater.cc:511] Master 127.29.159.126:33671 requested a full tablet report, sending...
I20260927 18:50:49.973773 17033 ts_manager.cc:194] Registered new tserver with Master: aa80f6e496294bee81069297e91026f9 (127.29.159.65:38621)
I20260927 18:50:49.975119 17033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.65@KRBTEST.COM'} at 127.29.159.65:41237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
Entry for principal kudu/127.29.159.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab.
I20260927 18:50:49.981978 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33671
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:50.095856 17229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:50.096115 17229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:50.096196 17229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:50.100400 17229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:50.100471 17229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:50.100579 17229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:50:50.105926 17229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.66.keytab
--principal=kudu/127.29.159.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:50.107033 17229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:50.108145 17229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:50.117942 17234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:50.117980 17229 server_base.cc:1034] running on GCE node
W20260927 18:50:50.118105 17235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:50.117942 17237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:50.118599 17229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:50.119185 17229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:50.120337 17229 hybrid_clock.cc:648] HybridClock initialized: now 1790535050120301 us; error 50 us; skew 500 ppm
Sep 27 18:50:50 dist-test-slave-8tg2 krb5kdc[16988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:50.123265 17229 init.cc:377] Logged in from keytab as kudu/127.29.159.66@KRBTEST.COM (short username kudu)
I20260927 18:50:50.124261 17229 webserver.cc:460] Webserver started at http://127.29.159.66:41785/ using document root <none> and password file <none>
I20260927 18:50:50.124835 17229 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:50.124897 17229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:50.125136 17229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:50.127146 17229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "f6fb3e5a9300462da7f8c3c58d04c55e"
format_stamp: "Formatted at 2026-09-27 18:50:50 on dist-test-slave-8tg2"
server_key: "3146e9c2f9af32e36fedd5cd1c22ace9"
server_key_iv: "e47452080c25b18c5ad21fe3cefec4f1"
server_key_version: "encryptionkey@0"
I20260927 18:50:50.127794 17229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "f6fb3e5a9300462da7f8c3c58d04c55e"
format_stamp: "Formatted at 2026-09-27 18:50:50 on dist-test-slave-8tg2"
server_key: "3146e9c2f9af32e36fedd5cd1c22ace9"
server_key_iv: "e47452080c25b18c5ad21fe3cefec4f1"
server_key_version: "encryptionkey@0"
I20260927 18:50:50.131560 17229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:50:50.133981 17244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:50.135243 17229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:50:50.135381 17229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "f6fb3e5a9300462da7f8c3c58d04c55e"
format_stamp: "Formatted at 2026-09-27 18:50:50 on dist-test-slave-8tg2"
server_key: "3146e9c2f9af32e36fedd5cd1c22ace9"
server_key_iv: "e47452080c25b18c5ad21fe3cefec4f1"
server_key_version: "encryptionkey@0"
I20260927 18:50:50.135574 17229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:50.152940 17229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:50.156323 17229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:50.156528 17229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:50.157128 17229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:50.157948 17229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:50.157999 17229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:50.158082 17229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:50.158119 17229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:50.167255 17229 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:35483
I20260927 18:50:50.167274 17357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:35483 every 8 connection(s)
I20260927 18:50:50.168222 17229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:50:50.168498 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 17229
I20260927 18:50:50.168645 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:50:50.169005 30333 external_mini_cluster.cc:1428] Setting key 1b6cc3e8d38518c945c7ffe7360886c3
Sep 27 18:50:50 dist-test-slave-8tg2 krb5kdc[16988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.66@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.67@KRBTEST.COM" created.
I20260927 18:50:50.182124 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.170175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:53811 (local address 127.29.159.126:33671)
0927 18:50:50.170317 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:50.170321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:50.171316 (+ 995us) server_negotiation.cc:407] Connection header received
0927 18:50:50.172724 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:50.172727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:50.172776 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:50.172854 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:50.173978 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.174497 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.175676 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.175896 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.176008 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.178419 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:50.178423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:50.178424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:50.178447 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:50.179960 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:50.180480 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:50.180487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:50.180489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:50.180571 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:50.180913 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:50.180919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:50.180924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:50.181189 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:50.181300 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:50.181824 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 18:50:50.181959 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:50:50.182971 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.170687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:33671 (local address 127.29.159.66:53811)
0927 18:50:50.171178 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:50.171219 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:50.172513 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:50.173033 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:50.173040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:50.173377 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:50.173837 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.173851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.174620 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.174624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.175557 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.175567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.176041 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.176045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.176218 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.176812 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:50.176828 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:50.178221 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:50.180091 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:50.180097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:50.180114 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:50.180369 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:50.180672 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:50.180675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:50.180677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:50.180817 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:50.181311 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:50.181320 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:50.181585 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 18:50:50.181783 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 18:50:50.182121 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":142,"threads_started":1}
I20260927 18:50:50.184319 17358 heartbeater.cc:347] Connected to a master server at 127.29.159.126:33671
I20260927 18:50:50.184612 17358 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:50.185137 17358 heartbeater.cc:511] Master 127.29.159.126:33671 requested a full tablet report, sending...
I20260927 18:50:50.186383 17033 ts_manager.cc:194] Registered new tserver with Master: f6fb3e5a9300462da7f8c3c58d04c55e (127.29.159.66:35483)
I20260927 18:50:50.187233 17033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.66@KRBTEST.COM'} at 127.29.159.66:53811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
Entry for principal kudu/127.29.159.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab.
I20260927 18:50:50.198755 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33671
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:50.310338 17363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:50.310599 17363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:50.310643 17363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:50.314805 17363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:50.314864 17363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:50.314922 17363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:50:50.320312 17363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.67.keytab
--principal=kudu/127.29.159.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:33671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:50.321359 17363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:50.322352 17363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:50.332300 17368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:50.332311 17371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:50.332311 17369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:50.332639 17363 server_base.cc:1034] running on GCE node
I20260927 18:50:50.333200 17363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:50.333833 17363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:50.335000 17363 hybrid_clock.cc:648] HybridClock initialized: now 1790535050334988 us; error 29 us; skew 500 ppm
Sep 27 18:50:50 dist-test-slave-8tg2 krb5kdc[16988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:50.337764 17363 init.cc:377] Logged in from keytab as kudu/127.29.159.67@KRBTEST.COM (short username kudu)
I20260927 18:50:50.338742 17363 webserver.cc:460] Webserver started at http://127.29.159.67:36859/ using document root <none> and password file <none>
I20260927 18:50:50.339299 17363 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:50.339387 17363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:50.339655 17363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:50.341740 17363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc"
format_stamp: "Formatted at 2026-09-27 18:50:50 on dist-test-slave-8tg2"
server_key: "8a7d5875a77411ed81e9d5aab11208db"
server_key_iv: "7737ae7581e35ea7c8f9d4fdbd0bd96b"
server_key_version: "encryptionkey@0"
I20260927 18:50:50.342330 17363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc"
format_stamp: "Formatted at 2026-09-27 18:50:50 on dist-test-slave-8tg2"
server_key: "8a7d5875a77411ed81e9d5aab11208db"
server_key_iv: "7737ae7581e35ea7c8f9d4fdbd0bd96b"
server_key_version: "encryptionkey@0"
I20260927 18:50:50.346086 17363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:50:50.348441 17378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:50.349603 17363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:50.349733 17363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc"
format_stamp: "Formatted at 2026-09-27 18:50:50 on dist-test-slave-8tg2"
server_key: "8a7d5875a77411ed81e9d5aab11208db"
server_key_iv: "7737ae7581e35ea7c8f9d4fdbd0bd96b"
server_key_version: "encryptionkey@0"
I20260927 18:50:50.349839 17363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:50.373121 17363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:50.376348 17363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:50.376549 17363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:50.377121 17363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:50.377916 17363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:50.377992 17363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:50.378063 17363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:50.378098 17363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:50.386605 17363 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:42755
I20260927 18:50:50.386632 17491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:42755 every 8 connection(s)
I20260927 18:50:50.387475 17363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:50:50.395341 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 17363
I20260927 18:50:50.395501 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:50:50.395828 30333 external_mini_cluster.cc:1428] Setting key a057725f8d5e3bc7abc3ff809b3822f1
Sep 27 18:50:50 dist-test-slave-8tg2 krb5kdc[16988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.67@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:50:50.401408 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.389435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:51677 (local address 127.29.159.126:33671)
0927 18:50:50.389574 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:50.389578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:50.390437 (+ 859us) server_negotiation.cc:407] Connection header received
0927 18:50:50.391999 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:50.392003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:50.392063 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:50.392169 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:50.393354 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.393903 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.395067 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.395286 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.395405 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.398062 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:50.398065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:50.398067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:50.398090 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:50.399468 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:50.399966 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:50.399968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:50.399969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:50.400066 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:50.400476 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:50.400479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:50.400480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:50.400646 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:50.400721 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:50.401197 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 18:50:50.401292 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:50:50.402256 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.389795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:33671 (local address 127.29.159.67:51677)
0927 18:50:50.390231 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:50.390272 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:50.391780 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:50.392349 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:50.392357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:50.392753 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:50.393213 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.393226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.394027 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.394031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.394938 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.394947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.395473 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.395478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.395687 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.396286 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:50:50.396302 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:50.397836 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:50.399590 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:50.399594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:50.399607 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:50.399878 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:50.400191 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:50:50.400194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:50:50.400196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:50:50.400385 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:50:50.400740 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:50.400745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:50.400984 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 18:50:50.401146 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 18:50:50.401478 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":104,"threads_started":1}
I20260927 18:50:50.403414 17492 heartbeater.cc:347] Connected to a master server at 127.29.159.126:33671
I20260927 18:50:50.403678 17492 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:50.404172 17492 heartbeater.cc:511] Master 127.29.159.126:33671 requested a full tablet report, sending...
I20260927 18:50:50.405190 17033 ts_manager.cc:194] Registered new tserver with Master: bac5a9e9c3654b64a7a89f7cba08d7bc (127.29.159.67:42755)
I20260927 18:50:50.405813 17033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.159.67@KRBTEST.COM'} at 127.29.159.67:51677
I20260927 18:50:50.409721 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:50:50.419315 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.411754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.29.159.126:33671)
0927 18:50:50.411915 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:50.411919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:50.412149 (+ 230us) server_negotiation.cc:407] Connection header received
0927 18:50:50.412211 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:50.412213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:50.412251 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:50.412321 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:50.412974 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.413557 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.414393 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.414600 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.414683 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.416027 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:50.416030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:50.416032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:50.416053 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:50.417563 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:50.417954 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:50.417958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:50.417960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:50.418053 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:50.418342 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:50.418347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:50.418349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:50.418654 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:50.418806 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:50.419036 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 18:50:50.419153 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:50:50.422372 17033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:50:50.425583 17033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:50:50.439924 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.434779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.29.159.67:42755)
0927 18:50:50.435164 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:50.435175 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:50.435200 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:50:50.435283 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:50.435289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:50.435544 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:50.435702 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:50.436030 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.436902 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.438444 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.439219 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.439341 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.439575 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:50.439677 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 18:50:50.439769 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":92,"threads_started":1}
I20260927 18:50:50.439985 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.434040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:42755 (local address 127.0.0.1:47530)
0927 18:50:50.434912 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:50.434965 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:50.435086 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:50.435709 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:50.435714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:50.435737 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:50.435896 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.435906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.437041 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.437048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.438294 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.438324 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.439352 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.439355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.439482 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.439527 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:50:50.439648 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 18:50:50.439769 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260927 18:50:50.440558 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.434325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:38621 (local address 127.0.0.1:36616)
0927 18:50:50.435331 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:50.435344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:50.435471 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:50.436119 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:50.436122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:50.436140 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:50.436222 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.436232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.437570 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.437573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.438760 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.438813 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.440143 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.440147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.440242 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.440262 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 18:50:50.440333 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:50:50.440403 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":56,"thread_start_us":77,"threads_started":1}
I20260927 18:50:50.440711 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.433990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:35483 (local address 127.0.0.1:47466)
0927 18:50:50.435250 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:50.435262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:50.435419 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:50.435938 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:50.435944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:50.435965 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:50.436060 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.436067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.437649 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.437652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.438699 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.438706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.440333 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.440336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.440427 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.440450 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:50:50.440526 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 18:50:50.440592 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260927 18:50:50.441115 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.434117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.29.159.66:35483)
0927 18:50:50.434540 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:50.434556 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:50.435485 (+ 929us) server_negotiation.cc:407] Connection header received
0927 18:50:50.435577 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:50.435581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:50.435782 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:50.435943 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:50.436179 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.437391 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.438812 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.439848 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.439936 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.440443 (+ 507us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:50.440573 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 18:50:50.440987 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":107,"threads_started":1}
I20260927 18:50:50.441169 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.434779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.29.159.65:38621)
0927 18:50:50.435205 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:50.435218 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:50.435576 (+ 358us) server_negotiation.cc:407] Connection header received
0927 18:50:50.435658 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:50.435663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:50.435843 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:50.436015 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:50.436348 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.437434 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.438927 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.439800 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.439927 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.440071 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:50.440948 (+ 877us) server_negotiation.cc:299] Negotiation successful
0927 18:50:50.441038 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":92,"threads_started":1}
I20260927 18:50:50.443181 17426 tablet_service.cc:1467] Processing CreateTablet for tablet 4db50ac853044460877f7f7e8ce50ac2 (DEFAULT_TABLE table=test-table [id=acfbde04a4c04c319f3e7a4b2f4db6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:50.443670 17292 tablet_service.cc:1467] Processing CreateTablet for tablet 4db50ac853044460877f7f7e8ce50ac2 (DEFAULT_TABLE table=test-table [id=acfbde04a4c04c319f3e7a4b2f4db6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:50.443863 17158 tablet_service.cc:1467] Processing CreateTablet for tablet 4db50ac853044460877f7f7e8ce50ac2 (DEFAULT_TABLE table=test-table [id=acfbde04a4c04c319f3e7a4b2f4db6b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:50.444335 17426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db50ac853044460877f7f7e8ce50ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:50.444669 17292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db50ac853044460877f7f7e8ce50ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:50.444860 17158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db50ac853044460877f7f7e8ce50ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:50.450593 17509 tablet_bootstrap.cc:492] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e: Bootstrap starting.
I20260927 18:50:50.452006 17511 tablet_bootstrap.cc:492] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc: Bootstrap starting.
I20260927 18:50:50.453362 17509 tablet_bootstrap.cc:654] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:50.453624 17510 tablet_bootstrap.cc:492] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9: Bootstrap starting.
I20260927 18:50:50.454195 17511 tablet_bootstrap.cc:654] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:50.454732 17509 log.cc:824] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:50.457180 17509 tablet_bootstrap.cc:492] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e: No bootstrap required, opened a new log
I20260927 18:50:50.457396 17509 ts_tablet_manager.cc:1397] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260927 18:50:50.455453 17511 log.cc:824] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:50.461366 17509 raft_consensus.cc:348] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35483 } } peers { permanent_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42755 } } peers { permanent_uuid: "aa80f6e496294bee81069297e91026f9" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38621 } }
I20260927 18:50:50.461568 17509 raft_consensus.cc:374] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:50.461601 17509 raft_consensus.cc:729] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6fb3e5a9300462da7f8c3c58d04c55e, State: Initialized, Role: FOLLOWER
I20260927 18:50:50.461700 17510 tablet_bootstrap.cc:654] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:50.461999 17509 consensus_queue.cc:260] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35483 } } peers { permanent_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42755 } } peers { permanent_uuid: "aa80f6e496294bee81069297e91026f9" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38621 } }
I20260927 18:50:50.463071 17510 log.cc:824] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:50.463734 17509 ts_tablet_manager.cc:1428] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260927 18:50:50.464260 17358 heartbeater.cc:503] Master 127.29.159.126:33671 was elected leader, sending a full tablet report...
I20260927 18:50:50.465576 17510 tablet_bootstrap.cc:492] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9: No bootstrap required, opened a new log
I20260927 18:50:50.465783 17510 ts_tablet_manager.cc:1397] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20260927 18:50:50.466022 17511 tablet_bootstrap.cc:492] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc: No bootstrap required, opened a new log
I20260927 18:50:50.466231 17511 ts_tablet_manager.cc:1397] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260927 18:50:50.469919 17511 raft_consensus.cc:348] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35483 } } peers { permanent_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42755 } } peers { permanent_uuid: "aa80f6e496294bee81069297e91026f9" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38621 } }
I20260927 18:50:50.470173 17511 raft_consensus.cc:374] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:50.470227 17511 raft_consensus.cc:729] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac5a9e9c3654b64a7a89f7cba08d7bc, State: Initialized, Role: FOLLOWER
I20260927 18:50:50.470670 17511 consensus_queue.cc:260] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35483 } } peers { permanent_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42755 } } peers { permanent_uuid: "aa80f6e496294bee81069297e91026f9" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38621 } }
I20260927 18:50:50.470762 17510 raft_consensus.cc:348] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35483 } } peers { permanent_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42755 } } peers { permanent_uuid: "aa80f6e496294bee81069297e91026f9" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38621 } }
I20260927 18:50:50.471027 17510 raft_consensus.cc:374] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:50.471097 17510 raft_consensus.cc:729] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa80f6e496294bee81069297e91026f9, State: Initialized, Role: FOLLOWER
I20260927 18:50:50.471479 17492 heartbeater.cc:503] Master 127.29.159.126:33671 was elected leader, sending a full tablet report...
I20260927 18:50:50.471591 17510 consensus_queue.cc:260] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35483 } } peers { permanent_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42755 } } peers { permanent_uuid: "aa80f6e496294bee81069297e91026f9" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38621 } }
I20260927 18:50:50.471720 17511 ts_tablet_manager.cc:1428] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260927 18:50:50.472359 17224 heartbeater.cc:503] Master 127.29.159.126:33671 was elected leader, sending a full tablet report...
I20260927 18:50:50.472560 17510 ts_tablet_manager.cc:1428] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 18:50:50.519848 17517 raft_consensus.cc:482] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:50.520077 17517 raft_consensus.cc:504] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35483 } } peers { permanent_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42755 } } peers { permanent_uuid: "aa80f6e496294bee81069297e91026f9" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38621 } }
I20260927 18:50:50.521304 17517 leader_election.cc:288] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6fb3e5a9300462da7f8c3c58d04c55e (127.29.159.66:35483), bac5a9e9c3654b64a7a89f7cba08d7bc (127.29.159.67:42755)
I20260927 18:50:50.526144 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.521692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:34045 (local address 127.29.159.67:42755)
0927 18:50:50.521851 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:50.521855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:50.522012 (+ 157us) server_negotiation.cc:407] Connection header received
0927 18:50:50.522229 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:50.522232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:50.522284 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:50.522335 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:50.522777 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.523824 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.525046 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.525474 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.525609 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.525671 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:50.525951 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 18:50:50.526027 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:50:50.526166 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.521530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:42755 (local address 127.29.159.65:34045)
0927 18:50:50.521923 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:50.521938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:50.522096 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:50.522529 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:50.522532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:50.522554 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:50.522653 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.522658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.523972 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.523975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.524922 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.524931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.525604 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.525608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.525820 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.525841 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 18:50:50.525932 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 18:50:50.526007 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":91,"spinlock_wait_cycles":768,"thread_start_us":125,"threads_started":1}
I20260927 18:50:50.527148 17446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db50ac853044460877f7f7e8ce50ac2" candidate_uuid: "aa80f6e496294bee81069297e91026f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" is_pre_election: true
I20260927 18:50:50.527477 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.522030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:60663 (local address 127.29.159.66:35483)
0927 18:50:50.522177 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:50.522184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:50.522199 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:50:50.522263 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:50.522268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:50.522332 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:50.522429 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:50.523524 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.524104 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.526012 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.526813 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.526963 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.527044 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:50.527162 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 18:50:50.527281 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:50:50.527508 17446 raft_consensus.cc:2454] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa80f6e496294bee81069297e91026f9 in term 0.
I20260927 18:50:50.527500 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.521549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:35483 (local address 127.29.159.65:60663)
0927 18:50:50.522044 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:50.522056 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:50.522126 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:50.522461 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:50.522466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:50.522491 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:50.522646 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.522653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.524238 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.524244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.525831 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:50.525846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.526934 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.526937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:50.527033 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.527055 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 18:50:50.527133 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:50:50.527209 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":31,"thread_start_us":139,"threads_started":1}
I20260927 18:50:50.528059 17112 leader_election.cc:302] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa80f6e496294bee81069297e91026f9, bac5a9e9c3654b64a7a89f7cba08d7bc; no voters:
I20260927 18:50:50.528350 17517 raft_consensus.cc:2781] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:50:50.528445 17517 raft_consensus.cc:482] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:50:50.528488 17517 raft_consensus.cc:3037] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:50.528458 17312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db50ac853044460877f7f7e8ce50ac2" candidate_uuid: "aa80f6e496294bee81069297e91026f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" is_pre_election: true
I20260927 18:50:50.528744 17312 raft_consensus.cc:2454] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa80f6e496294bee81069297e91026f9 in term 0.
I20260927 18:50:50.529691 17517 raft_consensus.cc:504] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35483 } } peers { permanent_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42755 } } peers { permanent_uuid: "aa80f6e496294bee81069297e91026f9" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38621 } }
I20260927 18:50:50.530015 17517 leader_election.cc:288] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [CANDIDATE]: Term 1 election: Requested vote from peers f6fb3e5a9300462da7f8c3c58d04c55e (127.29.159.66:35483), bac5a9e9c3654b64a7a89f7cba08d7bc (127.29.159.67:42755)
I20260927 18:50:50.530380 17312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db50ac853044460877f7f7e8ce50ac2" candidate_uuid: "aa80f6e496294bee81069297e91026f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e"
I20260927 18:50:50.530524 17312 raft_consensus.cc:3037] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:50.531600 17446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db50ac853044460877f7f7e8ce50ac2" candidate_uuid: "aa80f6e496294bee81069297e91026f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc"
I20260927 18:50:50.531744 17446 raft_consensus.cc:3037] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:50.532235 17312 raft_consensus.cc:2454] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa80f6e496294bee81069297e91026f9 in term 1.
I20260927 18:50:50.532565 17111 leader_election.cc:302] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa80f6e496294bee81069297e91026f9, f6fb3e5a9300462da7f8c3c58d04c55e; no voters:
I20260927 18:50:50.532768 17517 raft_consensus.cc:2781] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:50.533018 17517 raft_consensus.cc:686] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [term 1 LEADER]: Becoming Leader. State: Replica: aa80f6e496294bee81069297e91026f9, State: Running, Role: LEADER
I20260927 18:50:50.533215 17446 raft_consensus.cc:2454] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa80f6e496294bee81069297e91026f9 in term 1.
I20260927 18:50:50.533420 17517 consensus_queue.cc:237] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35483 } } peers { permanent_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42755 } } peers { permanent_uuid: "aa80f6e496294bee81069297e91026f9" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38621 } }
I20260927 18:50:50.537143 17033 catalog_manager.cc:5495] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa80f6e496294bee81069297e91026f9 (127.29.159.65). New cstate: current_term: 1 leader_uuid: "aa80f6e496294bee81069297e91026f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa80f6e496294bee81069297e91026f9" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 38621 } health_report { overall_health: HEALTHY } } }
I20260927 18:50:50.571193 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.566317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.29.159.65:38621)
0927 18:50:50.566466 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:50.566472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:50.566489 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:50:50.566590 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:50.566595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:50.566681 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:50.566818 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:50.567112 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.568072 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.569116 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.569557 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.569717 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.570002 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:50.570669 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:50.570815 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:50.570899 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 18:50:50.570989 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:50:50.577066 17312 raft_consensus.cc:1261] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e [term 1 FOLLOWER]: Refusing update from remote peer aa80f6e496294bee81069297e91026f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:50.577205 17446 raft_consensus.cc:1261] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc [term 1 FOLLOWER]: Refusing update from remote peer aa80f6e496294bee81069297e91026f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:50.577893 17520 consensus_queue.cc:1035] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac5a9e9c3654b64a7a89f7cba08d7bc" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:50.578094 17517 consensus_queue.cc:1035] T 4db50ac853044460877f7f7e8ce50ac2 P aa80f6e496294bee81069297e91026f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6fb3e5a9300462da7f8c3c58d04c55e" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 35483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:50.587378 17523 mvcc.cc:205] Tried to move back new op lower bound from 7334031567156162560 to 7334031566988402688. Current Snapshot: MvccSnapshot[applied={T|T < 7334031567156162560}]
I20260927 18:50:50.591029 17525 mvcc.cc:205] Tried to move back new op lower bound from 7334031567156162560 to 7334031566988402688. Current Snapshot: MvccSnapshot[applied={T|T < 7334031567156162560 or (T in {7334031567156162560})}]
I20260927 18:50:50.600687 17033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:50.600883 17033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:50.603896 17033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae264e60120d46bb8e90d05c1351df70: Sending DeleteTablet for 3 replicas of tablet 4db50ac853044460877f7f7e8ce50ac2
I20260927 18:50:50.604813 17292 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db50ac853044460877f7f7e8ce50ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:50 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:47466
I20260927 18:50:50.605178 17426 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db50ac853044460877f7f7e8ce50ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:50 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:47530
I20260927 18:50:50.605531 17533 tablet_replica.cc:330] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e: stopping tablet replica
I20260927 18:50:50.605700 17158 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db50ac853044460877f7f7e8ce50ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:50 UTC) from {username='kudu', principal='kudu/127.29.159.126@KRBTEST.COM'} at 127.0.0.1:36616
I20260927 18:50:50.606043 17533 raft_consensus.cc:2229] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:50.606451 17533 raft_consensus.cc:2258] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:50.607014 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 17095
I20260927 18:50:50.608311 17533 ts_tablet_manager.cc:1905] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:50.609954 17534 tablet_replica.cc:330] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc: stopping tablet replica
I20260927 18:50:50.610345 17534 raft_consensus.cc:2229] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:50.610679 17534 raft_consensus.cc:2258] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:50.612082 17533 ts_tablet_manager.cc:1918] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:50.612203 17533 log.cc:1196] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/4db50ac853044460877f7f7e8ce50ac2
I20260927 18:50:50.612618 17533 ts_tablet_manager.cc:1939] T 4db50ac853044460877f7f7e8ce50ac2 P f6fb3e5a9300462da7f8c3c58d04c55e: Deleting consensus metadata
I20260927 18:50:50.614208 17017 catalog_manager.cc:4841] TS f6fb3e5a9300462da7f8c3c58d04c55e (127.29.159.66:35483): tablet 4db50ac853044460877f7f7e8ce50ac2 (table test-table [id=acfbde04a4c04c319f3e7a4b2f4db6b4]) successfully deleted
W20260927 18:50:50.616339 17020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:38621 (error 108)
I20260927 18:50:50.616420 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 17229
I20260927 18:50:50.618698 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.618201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:38621 (local address 127.0.0.1:36622)
0927 18:50:50.618465 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:50.618583 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:38621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260927 18:50:50.619062 17020 catalog_manager.cc:4584] TS aa80f6e496294bee81069297e91026f9 (127.29.159.65:38621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4db50ac853044460877f7f7e8ce50ac2: Network error: Client connection negotiation failed: client connection to 127.29.159.65:38621: connect: Connection refused (error 111)
I20260927 18:50:50.619316 17534 ts_tablet_manager.cc:1905] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:50.621716 17534 ts_tablet_manager.cc:1918] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:50.621852 17534 log.cc:1196] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/4db50ac853044460877f7f7e8ce50ac2
I20260927 18:50:50.622265 17534 ts_tablet_manager.cc:1939] T 4db50ac853044460877f7f7e8ce50ac2 P bac5a9e9c3654b64a7a89f7cba08d7bc: Deleting consensus metadata
I20260927 18:50:50.623973 17018 catalog_manager.cc:4841] TS bac5a9e9c3654b64a7a89f7cba08d7bc (127.29.159.67:42755): tablet 4db50ac853044460877f7f7e8ce50ac2 (table test-table [id=acfbde04a4c04c319f3e7a4b2f4db6b4]) successfully deleted
I20260927 18:50:50.627152 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 17363
I20260927 18:50:50.633054 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 17001
2026-09-27T18:50:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2312 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T18:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:50Z Disabled control of system clock
I20260927 18:50:50.663864 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:43955
--webserver_interface=127.29.159.126
--webserver_port=0
--builtin_ntp_servers=127.29.159.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:43955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:50:50.776196 17540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:50.776463 17540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:50.776497 17540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:50.780511 17540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:50.780566 17540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:50.780582 17540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:50.780598 17540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:50.780613 17540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:50.785861 17540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:43955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:43955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:50.786942 17540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:50.788393 17540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:50:50.796617 17540 server_base.cc:1034] running on GCE node
W20260927 18:50:50.796588 17545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:50.796577 17548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:50.796913 17546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:50.797433 17540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:50.798506 17540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:50.799659 17540 hybrid_clock.cc:648] HybridClock initialized: now 1790535050799647 us; error 29 us; skew 500 ppm
I20260927 18:50:50.801499 17540 webserver.cc:460] Webserver started at http://127.29.159.126:37947/ using document root <none> and password file <none>
I20260927 18:50:50.802084 17540 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:50.802146 17540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:50.802358 17540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:50.804507 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "f5f811c20383484c8ba5f7e39ad2668d"
format_stamp: "Formatted at 2026-09-27 18:50:50 on dist-test-slave-8tg2"
server_key: "68c58214e237b809a254e639fcc81a7a"
server_key_iv: "ebbb31d2047d8895537d7bc79ad13fb0"
server_key_version: "encryptionkey@0"
I20260927 18:50:50.805136 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "f5f811c20383484c8ba5f7e39ad2668d"
format_stamp: "Formatted at 2026-09-27 18:50:50 on dist-test-slave-8tg2"
server_key: "68c58214e237b809a254e639fcc81a7a"
server_key_iv: "ebbb31d2047d8895537d7bc79ad13fb0"
server_key_version: "encryptionkey@0"
I20260927 18:50:50.809068 17540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 18:50:50.811637 17554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:50.812867 17540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:50:50.813026 17540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "f5f811c20383484c8ba5f7e39ad2668d"
format_stamp: "Formatted at 2026-09-27 18:50:50 on dist-test-slave-8tg2"
server_key: "68c58214e237b809a254e639fcc81a7a"
server_key_iv: "ebbb31d2047d8895537d7bc79ad13fb0"
server_key_version: "encryptionkey@0"
I20260927 18:50:50.813148 17540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:50.856493 17540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:50.857266 17540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:50.857481 17540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:50.864482 17540 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:43955
I20260927 18:50:50.864502 17606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:43955 every 8 connection(s)
I20260927 18:50:50.865435 17540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:50.868561 17607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:50.869876 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 17540
I20260927 18:50:50.870023 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:50:50.870357 30333 external_mini_cluster.cc:1428] Setting key 42efa83ec81d9223887ecc13d6e23050
I20260927 18:50:50.874722 17607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d: Bootstrap starting.
I20260927 18:50:50.876958 17607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:50.877683 17607 log.cc:824] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:50.879601 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:50.871928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.29.159.126:43955)
0927 18:50:50.872576 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:50.872591 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:50.872626 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:50:50.873257 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:50.873274 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:50.873664 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:50.874001 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:50.874427 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.875485 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.876376 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:50.876844 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:50.877023 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:50.877878 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:50.877883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:50.877896 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:50.878269 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:50.878338 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:50.878358 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:50.878474 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:50.878596 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 18:50:50.878845 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
I20260927 18:50:50.879707 17607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d: No bootstrap required, opened a new log
I20260927 18:50:50.883149 17607 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f811c20383484c8ba5f7e39ad2668d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43955 } }
I20260927 18:50:50.883396 17607 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:50.883493 17607 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f811c20383484c8ba5f7e39ad2668d, State: Initialized, Role: FOLLOWER
I20260927 18:50:50.883937 17607 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f811c20383484c8ba5f7e39ad2668d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43955 } }
I20260927 18:50:50.884087 17607 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:50.884174 17607 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:50.884271 17607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:50.885288 17607 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f811c20383484c8ba5f7e39ad2668d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43955 } }
I20260927 18:50:50.885649 17607 leader_election.cc:302] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5f811c20383484c8ba5f7e39ad2668d; no voters:
I20260927 18:50:50.885964 17607 leader_election.cc:288] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:50.886086 17612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:50.886336 17612 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [term 1 LEADER]: Becoming Leader. State: Replica: f5f811c20383484c8ba5f7e39ad2668d, State: Running, Role: LEADER
I20260927 18:50:50.886631 17612 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f811c20383484c8ba5f7e39ad2668d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43955 } }
I20260927 18:50:50.887154 17607 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:50.888312 17614 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5f811c20383484c8ba5f7e39ad2668d. Latest consensus state: current_term: 1 leader_uuid: "f5f811c20383484c8ba5f7e39ad2668d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f811c20383484c8ba5f7e39ad2668d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43955 } } }
I20260927 18:50:50.888432 17614 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:50.888717 17613 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5f811c20383484c8ba5f7e39ad2668d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f811c20383484c8ba5f7e39ad2668d" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43955 } } }
I20260927 18:50:50.888806 17613 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:50.889101 17621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:50.892315 17621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:50.897881 17621 catalog_manager.cc:1347] Generated new cluster ID: 00b0af2374ec44529bc071b23dd0c65a
I20260927 18:50:50.897953 17621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:50.906855 17621 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:50.908067 17621 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:50.912611 17621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d: Generated new TSK 0
I20260927 18:50:50.913239 17621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:50.918292 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.65:0
--local_ip_for_outbound_sockets=127.29.159.65
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43955
--builtin_ntp_servers=127.29.159.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:50:51.035405 17631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:51.035725 17631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:51.035768 17631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:51.039897 17631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:51.039959 17631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:51.040023 17631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.65
I20260927 18:50:51.044976 17631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.159.65
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.65
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:51.046069 17631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:51.047549 17631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:51.057080 17636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:51.057094 17637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:51.057094 17639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:51.057467 17631 server_base.cc:1034] running on GCE node
I20260927 18:50:51.057866 17631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:51.058525 17631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:51.059706 17631 hybrid_clock.cc:648] HybridClock initialized: now 1790535051059688 us; error 38 us; skew 500 ppm
I20260927 18:50:51.061530 17631 webserver.cc:460] Webserver started at http://127.29.159.65:39203/ using document root <none> and password file <none>
I20260927 18:50:51.062114 17631 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:51.062206 17631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:51.062443 17631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:51.064527 17631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/data/instance:
uuid: "b61b2310370b4c878bca741b0ca025f7"
format_stamp: "Formatted at 2026-09-27 18:50:51 on dist-test-slave-8tg2"
server_key: "15a8482da8ec9b1fad089980aeed7d68"
server_key_iv: "657bef9bcdb477a63ad085fab0f583eb"
server_key_version: "encryptionkey@0"
I20260927 18:50:51.065088 17631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance:
uuid: "b61b2310370b4c878bca741b0ca025f7"
format_stamp: "Formatted at 2026-09-27 18:50:51 on dist-test-slave-8tg2"
server_key: "15a8482da8ec9b1fad089980aeed7d68"
server_key_iv: "657bef9bcdb477a63ad085fab0f583eb"
server_key_version: "encryptionkey@0"
I20260927 18:50:51.068749 17631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:51.071142 17645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:51.072413 17631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:50:51.072546 17631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/wal
uuid: "b61b2310370b4c878bca741b0ca025f7"
format_stamp: "Formatted at 2026-09-27 18:50:51 on dist-test-slave-8tg2"
server_key: "15a8482da8ec9b1fad089980aeed7d68"
server_key_iv: "657bef9bcdb477a63ad085fab0f583eb"
server_key_version: "encryptionkey@0"
I20260927 18:50:51.072660 17631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:51.085062 17631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:51.085762 17631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:51.085964 17631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:51.086539 17631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:51.087616 17631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:51.087682 17631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:51.087749 17631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:51.087800 17631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:51.096458 17631 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.65:46071
I20260927 18:50:51.096530 17758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.65:46071 every 8 connection(s)
I20260927 18:50:51.097517 17631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/data/info.pb
I20260927 18:50:51.105394 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 17631
I20260927 18:50:51.105551 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-0/wal/instance
I20260927 18:50:51.105893 30333 external_mini_cluster.cc:1428] Setting key 3f82620782c6b1358722b3aa84c75742
I20260927 18:50:51.107146 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.099551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.65:53425 (local address 127.29.159.126:43955)
0927 18:50:51.099698 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:51.099701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:51.100837 (+ 1136us) server_negotiation.cc:407] Connection header received
0927 18:50:51.102208 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:51.102211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:51.102265 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:51.102345 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:51.103255 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.103876 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.105011 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.105237 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.105321 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.106617 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:51.106620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:51.106623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:51.106636 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:51.106667 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:51.106675 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:51.106752 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:51.106926 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 18:50:51.107002 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:50:51.108054 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.66:0
--local_ip_for_outbound_sockets=127.29.159.66
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43955
--builtin_ntp_servers=127.29.159.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:50:51.108090 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.099997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:43955 (local address 127.29.159.65:53425)
0927 18:50:51.100661 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:51.100703 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:51.102015 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:51.102513 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:51.102519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:51.102610 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:51.103132 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.103145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.103997 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.104000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.104889 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.104898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.105409 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.105416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.105642 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.106323 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:50:51.106325 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:51.106385 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:50:51.106451 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:51.106766 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:51.106777 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:51.106782 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:50:51.106915 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 18:50:51.107215 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
I20260927 18:50:51.109494 17759 heartbeater.cc:347] Connected to a master server at 127.29.159.126:43955
I20260927 18:50:51.109728 17759 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:51.110275 17759 heartbeater.cc:511] Master 127.29.159.126:43955 requested a full tablet report, sending...
I20260927 18:50:51.111992 17571 ts_manager.cc:194] Registered new tserver with Master: b61b2310370b4c878bca741b0ca025f7 (127.29.159.65:46071)
I20260927 18:50:51.113765 17571 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.65:53425
W20260927 18:50:51.220486 17762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:51.220758 17762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:51.220800 17762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:51.225317 17762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:51.225378 17762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:51.225438 17762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.66
I20260927 18:50:51.230818 17762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.159.66
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.66
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:51.231948 17762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:51.233275 17762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:51.242631 17767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:51.242681 17768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:51.242802 17762 server_base.cc:1034] running on GCE node
W20260927 18:50:51.242626 17770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:51.243472 17762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:51.244253 17762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:51.245433 17762 hybrid_clock.cc:648] HybridClock initialized: now 1790535051245404 us; error 46 us; skew 500 ppm
I20260927 18:50:51.247674 17762 webserver.cc:460] Webserver started at http://127.29.159.66:39351/ using document root <none> and password file <none>
I20260927 18:50:51.248248 17762 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:51.248337 17762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:51.248564 17762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:51.250612 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/data/instance:
uuid: "c1b1518365134184bbf0ec6eb73479a2"
format_stamp: "Formatted at 2026-09-27 18:50:51 on dist-test-slave-8tg2"
server_key: "9f1c6ae59e0bd211b6b397226da9f532"
server_key_iv: "569a3b15ab113471cdc2be812e860a65"
server_key_version: "encryptionkey@0"
I20260927 18:50:51.251176 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance:
uuid: "c1b1518365134184bbf0ec6eb73479a2"
format_stamp: "Formatted at 2026-09-27 18:50:51 on dist-test-slave-8tg2"
server_key: "9f1c6ae59e0bd211b6b397226da9f532"
server_key_iv: "569a3b15ab113471cdc2be812e860a65"
server_key_version: "encryptionkey@0"
I20260927 18:50:51.254845 17762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:50:51.257351 17776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:51.258581 17762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:50:51.258711 17762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/wal
uuid: "c1b1518365134184bbf0ec6eb73479a2"
format_stamp: "Formatted at 2026-09-27 18:50:51 on dist-test-slave-8tg2"
server_key: "9f1c6ae59e0bd211b6b397226da9f532"
server_key_iv: "569a3b15ab113471cdc2be812e860a65"
server_key_version: "encryptionkey@0"
I20260927 18:50:51.258819 17762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:51.306732 17762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:51.307552 17762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:51.307749 17762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:51.308360 17762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:51.309324 17762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:51.309403 17762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:51.309477 17762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:51.309528 17762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:51.318683 17762 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.66:45459
I20260927 18:50:51.318702 17889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.66:45459 every 8 connection(s)
I20260927 18:50:51.319794 17762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/data/info.pb
I20260927 18:50:51.325470 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 17762
I20260927 18:50:51.325618 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/wal/instance
I20260927 18:50:51.325937 30333 external_mini_cluster.cc:1428] Setting key b53640cfb421f83b9c99bd084783df18
I20260927 18:50:51.328011 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.159.67:0
--local_ip_for_outbound_sockets=127.29.159.67
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43955
--builtin_ntp_servers=127.29.159.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:50:51.330083 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.321669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.66:45167 (local address 127.29.159.126:43955)
0927 18:50:51.321815 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:51.321820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:51.322997 (+ 1177us) server_negotiation.cc:407] Connection header received
0927 18:50:51.324498 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:51.324500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:51.324550 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:51.324625 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:51.325606 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.326285 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.327874 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.328092 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.328237 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.329549 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:51.329554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:51.329558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:51.329574 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:51.329600 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:51.329611 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:51.329693 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:51.329835 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 18:50:51.329918 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:50:51.330884 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.322159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:43955 (local address 127.29.159.66:45167)
0927 18:50:51.322840 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:51.322881 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:51.324274 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:51.324794 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:51.324801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:51.324908 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:51.325431 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.325444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.326469 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.326508 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.327717 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.327728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.328420 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.328424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.328567 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.329274 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:50:51.329277 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:51.329327 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:50:51.329388 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:51.329699 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:51.329705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:51.329709 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:50:51.329803 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 18:50:51.330023 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260927 18:50:51.332077 17890 heartbeater.cc:347] Connected to a master server at 127.29.159.126:43955
I20260927 18:50:51.332352 17890 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:51.332985 17890 heartbeater.cc:511] Master 127.29.159.126:43955 requested a full tablet report, sending...
I20260927 18:50:51.334097 17571 ts_manager.cc:194] Registered new tserver with Master: c1b1518365134184bbf0ec6eb73479a2 (127.29.159.66:45459)
I20260927 18:50:51.334659 17571 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.66:45167
W20260927 18:50:51.441208 17893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:51.441447 17893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:51.441485 17893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:51.445485 17893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:51.445533 17893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:51.445619 17893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.159.67
I20260927 18:50:51.450273 17893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.159.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.159.67
--webserver_port=0
--tserver_master_addrs=127.29.159.126:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.159.67
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:51.451304 17893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:51.452696 17893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:50:51.461905 17893 server_base.cc:1034] running on GCE node
W20260927 18:50:51.461900 17899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:51.461866 17898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:51.461930 17901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:51.462527 17893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:51.463181 17893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:51.464370 17893 hybrid_clock.cc:648] HybridClock initialized: now 1790535051464337 us; error 45 us; skew 500 ppm
I20260927 18:50:51.466233 17893 webserver.cc:460] Webserver started at http://127.29.159.67:39137/ using document root <none> and password file <none>
I20260927 18:50:51.466863 17893 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:51.466962 17893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:51.467199 17893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:51.469331 17893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/data/instance:
uuid: "5dc64e9212cb45f98900448483df54ed"
format_stamp: "Formatted at 2026-09-27 18:50:51 on dist-test-slave-8tg2"
server_key: "6e96146c6f5e0e3ffb6926477167a14f"
server_key_iv: "eb70df1a1804c159437e3914110b8567"
server_key_version: "encryptionkey@0"
I20260927 18:50:51.469926 17893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance:
uuid: "5dc64e9212cb45f98900448483df54ed"
format_stamp: "Formatted at 2026-09-27 18:50:51 on dist-test-slave-8tg2"
server_key: "6e96146c6f5e0e3ffb6926477167a14f"
server_key_iv: "eb70df1a1804c159437e3914110b8567"
server_key_version: "encryptionkey@0"
I20260927 18:50:51.473621 17893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:50:51.476188 17907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:51.477526 17893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:50:51.477666 17893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/wal
uuid: "5dc64e9212cb45f98900448483df54ed"
format_stamp: "Formatted at 2026-09-27 18:50:51 on dist-test-slave-8tg2"
server_key: "6e96146c6f5e0e3ffb6926477167a14f"
server_key_iv: "eb70df1a1804c159437e3914110b8567"
server_key_version: "encryptionkey@0"
I20260927 18:50:51.477794 17893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:51.506100 17893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:51.506865 17893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:51.507073 17893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:51.507755 17893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:50:51.508719 17893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:50:51.508790 17893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:51.508864 17893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:50:51.508908 17893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:51.518052 17893 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.67:45563
I20260927 18:50:51.518077 18020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.67:45563 every 8 connection(s)
I20260927 18:50:51.519004 17893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/data/info.pb
I20260927 18:50:51.524825 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 17893
I20260927 18:50:51.524931 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/wal/instance
I20260927 18:50:51.525198 30333 external_mini_cluster.cc:1428] Setting key 44bc3e4645742415d1430c6d5b4d8b65
I20260927 18:50:51.528506 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.520982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:43415 (local address 127.29.159.126:43955)
0927 18:50:51.521120 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:51.521124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:51.522192 (+ 1068us) server_negotiation.cc:407] Connection header received
0927 18:50:51.523594 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:51.523597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:51.523647 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:51.523725 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:51.524659 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.525321 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.526563 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.526819 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.526899 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.528112 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:51.528115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:51.528117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:51.528128 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:51.528145 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:51.528152 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:51.528215 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:51.528350 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 18:50:51.528398 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:50:51.529354 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.521383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:43955 (local address 127.29.159.67:43415)
0927 18:50:51.522029 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:51.522069 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:51.523368 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:51.523886 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:51.523893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:51.523980 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:50:51.524501 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.524515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.525446 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.525450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.526468 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.526478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.526993 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.526997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.527187 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.527864 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:50:51.527865 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:50:51.527913 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:50:51.527978 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:50:51.528233 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:50:51.528239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:50:51.528243 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 18:50:51.528336 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 18:50:51.528557 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260927 18:50:51.530380 18021 heartbeater.cc:347] Connected to a master server at 127.29.159.126:43955
I20260927 18:50:51.530639 18021 heartbeater.cc:464] Registering TS with master...
I20260927 18:50:51.531141 18021 heartbeater.cc:511] Master 127.29.159.126:43955 requested a full tablet report, sending...
I20260927 18:50:51.532174 17571 ts_manager.cc:194] Registered new tserver with Master: 5dc64e9212cb45f98900448483df54ed (127.29.159.67:45563)
I20260927 18:50:51.532727 17571 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.159.67:43415
I20260927 18:50:51.539304 30333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:50:51.545321 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.541283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.29.159.126:43955)
0927 18:50:51.541428 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:51.541434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:51.541635 (+ 201us) server_negotiation.cc:407] Connection header received
0927 18:50:51.541783 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:51.541787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:51.541835 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:51.541928 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:51.542298 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.542840 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.543784 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.544021 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.544131 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.544861 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:51.544865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:51.544867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:51.544879 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:51.544898 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:51.544914 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:51.545006 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:51.545125 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 18:50:51.545181 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:50:51.547978 17571 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:50:51.550835 17571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:50:51.565326 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.559051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:46071 (local address 127.0.0.1:33316)
0927 18:50:51.559878 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:51.559935 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:51.560058 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:51.560934 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:51.560938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:51.560954 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:51.561074 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.561081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.562487 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.562494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.563710 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.563719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.564776 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.564779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.564887 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.564932 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:50:51.565043 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 18:50:51.565166 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":25,"spinlock_wait_cycles":33152,"thread_start_us":101,"threads_started":1}
I20260927 18:50:51.565331 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.559420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.29.159.65:46071)
0927 18:50:51.560419 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:51.560431 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:51.560453 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:50:51.560541 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:51.560547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:51.560786 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:51.560943 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:51.561192 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.562356 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.563829 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.564657 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.564774 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.564980 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:51.565071 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 18:50:51.565168 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":145,"threads_started":1}
I20260927 18:50:51.565526 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.559768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.29.159.66:45459)
0927 18:50:51.560232 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:51.560242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:51.560432 (+ 190us) server_negotiation.cc:407] Connection header received
0927 18:50:51.560505 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:51.560510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:51.560706 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:51.560851 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:51.561241 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.562035 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.564001 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.564789 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.564888 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.565054 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:51.565250 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 18:50:51.565353 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":103,"threads_started":1}
I20260927 18:50:51.565855 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.559617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:45459 (local address 127.0.0.1:59478)
0927 18:50:51.560264 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:51.560275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:51.560348 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:51.561020 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:51.561023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:51.561038 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:51.561114 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.561123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.562159 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.562164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.563873 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.563878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.564928 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.564932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.565019 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.565047 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 18:50:51.565108 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 18:50:51.565166 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":221,"thread_start_us":80,"threads_started":1}
I20260927 18:50:51.566699 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.558965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.67:45563 (local address 127.0.0.1:57224)
0927 18:50:51.559878 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:51.559930 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:51.560058 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:51.560711 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:51.560717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:51.560748 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:51.560920 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.560928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.562443 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.562447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.563867 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.563878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.566292 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.566295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.566381 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.566417 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:50:51.566488 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:50:51.566556 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":89,"thread_start_us":74,"threads_started":1}
I20260927 18:50:51.567238 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.559096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.29.159.67:45563)
0927 18:50:51.559909 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:51.559921 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:51.560199 (+ 278us) server_negotiation.cc:407] Connection header received
0927 18:50:51.560283 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:51.560292 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:51.560520 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:51.560688 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:51.561066 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.562317 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.565113 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.565937 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.566047 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.566228 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:51.566967 (+ 739us) server_negotiation.cc:299] Negotiation successful
0927 18:50:51.567081 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":143,"threads_started":1}
I20260927 18:50:51.568660 17693 tablet_service.cc:1467] Processing CreateTablet for tablet cc7cad36dc654279b5f063fc7fdbe559 (DEFAULT_TABLE table=test-table [id=9013215d9ca6464c862ef7dd07b13020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:51.568660 17823 tablet_service.cc:1467] Processing CreateTablet for tablet cc7cad36dc654279b5f063fc7fdbe559 (DEFAULT_TABLE table=test-table [id=9013215d9ca6464c862ef7dd07b13020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:51.569588 17693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7cad36dc654279b5f063fc7fdbe559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:51.569597 17823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7cad36dc654279b5f063fc7fdbe559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:51.574470 17955 tablet_service.cc:1467] Processing CreateTablet for tablet cc7cad36dc654279b5f063fc7fdbe559 (DEFAULT_TABLE table=test-table [id=9013215d9ca6464c862ef7dd07b13020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:50:51.575896 17955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7cad36dc654279b5f063fc7fdbe559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:51.577025 18038 tablet_bootstrap.cc:492] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: Bootstrap starting.
I20260927 18:50:51.577716 18039 tablet_bootstrap.cc:492] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7: Bootstrap starting.
I20260927 18:50:51.579010 18038 tablet_bootstrap.cc:654] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:51.579818 18038 log.cc:824] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:51.579872 18039 tablet_bootstrap.cc:654] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:51.580691 18039 log.cc:824] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:51.581979 18038 tablet_bootstrap.cc:492] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: No bootstrap required, opened a new log
I20260927 18:50:51.582183 18038 ts_tablet_manager.cc:1397] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:50:51.582733 18039 tablet_bootstrap.cc:492] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7: No bootstrap required, opened a new log
I20260927 18:50:51.582913 18039 ts_tablet_manager.cc:1397] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:50:51.583482 18042 tablet_bootstrap.cc:492] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: Bootstrap starting.
I20260927 18:50:51.585685 18042 tablet_bootstrap.cc:654] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:51.586516 18042 log.cc:824] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:51.586426 18039 raft_consensus.cc:348] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc64e9212cb45f98900448483df54ed" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45563 } } peers { permanent_uuid: "b61b2310370b4c878bca741b0ca025f7" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46071 } } peers { permanent_uuid: "c1b1518365134184bbf0ec6eb73479a2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45459 } }
I20260927 18:50:51.586675 18039 raft_consensus.cc:374] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:51.586771 18039 raft_consensus.cc:729] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b61b2310370b4c878bca741b0ca025f7, State: Initialized, Role: FOLLOWER
I20260927 18:50:51.586582 18038 raft_consensus.cc:348] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc64e9212cb45f98900448483df54ed" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45563 } } peers { permanent_uuid: "b61b2310370b4c878bca741b0ca025f7" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46071 } } peers { permanent_uuid: "c1b1518365134184bbf0ec6eb73479a2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45459 } }
I20260927 18:50:51.586916 18038 raft_consensus.cc:374] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:51.586970 18038 raft_consensus.cc:729] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b1518365134184bbf0ec6eb73479a2, State: Initialized, Role: FOLLOWER
I20260927 18:50:51.587248 18039 consensus_queue.cc:260] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc64e9212cb45f98900448483df54ed" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45563 } } peers { permanent_uuid: "b61b2310370b4c878bca741b0ca025f7" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46071 } } peers { permanent_uuid: "c1b1518365134184bbf0ec6eb73479a2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45459 } }
I20260927 18:50:51.587404 18038 consensus_queue.cc:260] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc64e9212cb45f98900448483df54ed" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45563 } } peers { permanent_uuid: "b61b2310370b4c878bca741b0ca025f7" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46071 } } peers { permanent_uuid: "c1b1518365134184bbf0ec6eb73479a2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45459 } }
I20260927 18:50:51.588413 18039 ts_tablet_manager.cc:1428] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:50:51.588491 18038 ts_tablet_manager.cc:1428] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 18:50:51.588802 17759 heartbeater.cc:503] Master 127.29.159.126:43955 was elected leader, sending a full tablet report...
I20260927 18:50:51.588814 17890 heartbeater.cc:503] Master 127.29.159.126:43955 was elected leader, sending a full tablet report...
I20260927 18:50:51.589234 18042 tablet_bootstrap.cc:492] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: No bootstrap required, opened a new log
I20260927 18:50:51.589421 18042 ts_tablet_manager.cc:1397] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 18:50:51.593463 18042 raft_consensus.cc:348] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc64e9212cb45f98900448483df54ed" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45563 } } peers { permanent_uuid: "b61b2310370b4c878bca741b0ca025f7" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46071 } } peers { permanent_uuid: "c1b1518365134184bbf0ec6eb73479a2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45459 } }
I20260927 18:50:51.593670 18042 raft_consensus.cc:374] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:51.593757 18042 raft_consensus.cc:729] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc64e9212cb45f98900448483df54ed, State: Initialized, Role: FOLLOWER
I20260927 18:50:51.594143 18042 consensus_queue.cc:260] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc64e9212cb45f98900448483df54ed" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45563 } } peers { permanent_uuid: "b61b2310370b4c878bca741b0ca025f7" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46071 } } peers { permanent_uuid: "c1b1518365134184bbf0ec6eb73479a2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45459 } }
I20260927 18:50:51.594718 18021 heartbeater.cc:503] Master 127.29.159.126:43955 was elected leader, sending a full tablet report...
I20260927 18:50:51.595024 18042 ts_tablet_manager.cc:1428] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 18:50:51.598616 17760 tablet.cc:2369] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:50:51.748325 18046 raft_consensus.cc:482] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:50:51.748567 18046 raft_consensus.cc:504] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc64e9212cb45f98900448483df54ed" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45563 } } peers { permanent_uuid: "b61b2310370b4c878bca741b0ca025f7" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46071 } } peers { permanent_uuid: "c1b1518365134184bbf0ec6eb73479a2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45459 } }
I20260927 18:50:51.749776 18046 leader_election.cc:288] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b61b2310370b4c878bca741b0ca025f7 (127.29.159.65:46071), c1b1518365134184bbf0ec6eb73479a2 (127.29.159.66:45459)
I20260927 18:50:51.754949 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.750281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:49835 (local address 127.29.159.66:45459)
0927 18:50:51.750427 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:51.750431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:51.750854 (+ 423us) server_negotiation.cc:407] Connection header received
0927 18:50:51.750919 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:51.750922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:51.750993 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:51.751082 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:51.751395 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.752188 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.753622 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.754251 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.754390 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.754441 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:51.754740 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 18:50:51.754843 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:50:51.754948 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.750280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.159.67:37395 (local address 127.29.159.65:46071)
0927 18:50:51.750446 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:51.750451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:51.750470 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:50:51.750543 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:51.750549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:51.750607 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:51.750716 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:50:51.751036 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.751963 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.753622 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.754251 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.754390 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.754441 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:51.754740 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 18:50:51.754818 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:50:51.754966 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.750129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:46071 (local address 127.29.159.67:37395)
0927 18:50:51.750278 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:51.750296 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:51.750413 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:51.750729 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:51.750733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:51.750760 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:51.750897 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.750907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.752091 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.752095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.753422 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.753433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.754390 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.754394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.754556 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.754625 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 18:50:51.754702 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:50:51.754809 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":101}
I20260927 18:50:51.754983 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.750129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.66:45459 (local address 127.29.159.67:49835)
0927 18:50:51.750727 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:51.750741 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:50:51.750828 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:50:51.751106 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:50:51.751108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:50:51.751125 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:50:51.751218 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.751222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.752306 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.752309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.753446 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:50:51.753452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.754390 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.754394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:50:51.754570 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.754627 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 18:50:51.754702 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:50:51.754797 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":313,"thread_start_us":153,"threads_started":1}
I20260927 18:50:51.755820 17844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7cad36dc654279b5f063fc7fdbe559" candidate_uuid: "5dc64e9212cb45f98900448483df54ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b1518365134184bbf0ec6eb73479a2" is_pre_election: true
I20260927 18:50:51.755827 17713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7cad36dc654279b5f063fc7fdbe559" candidate_uuid: "5dc64e9212cb45f98900448483df54ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b61b2310370b4c878bca741b0ca025f7" is_pre_election: true
I20260927 18:50:51.756141 17713 raft_consensus.cc:2454] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dc64e9212cb45f98900448483df54ed in term 0.
I20260927 18:50:51.756134 17844 raft_consensus.cc:2454] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dc64e9212cb45f98900448483df54ed in term 0.
I20260927 18:50:51.756691 17909 leader_election.cc:302] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5dc64e9212cb45f98900448483df54ed, b61b2310370b4c878bca741b0ca025f7, c1b1518365134184bbf0ec6eb73479a2; no voters:
I20260927 18:50:51.757011 18046 raft_consensus.cc:2781] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:50:51.757115 18046 raft_consensus.cc:482] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:50:51.757179 18046 raft_consensus.cc:3037] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:51.758206 18046 raft_consensus.cc:504] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc64e9212cb45f98900448483df54ed" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45563 } } peers { permanent_uuid: "b61b2310370b4c878bca741b0ca025f7" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46071 } } peers { permanent_uuid: "c1b1518365134184bbf0ec6eb73479a2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45459 } }
I20260927 18:50:51.758623 18046 leader_election.cc:288] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [CANDIDATE]: Term 1 election: Requested vote from peers b61b2310370b4c878bca741b0ca025f7 (127.29.159.65:46071), c1b1518365134184bbf0ec6eb73479a2 (127.29.159.66:45459)
I20260927 18:50:51.758935 17713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7cad36dc654279b5f063fc7fdbe559" candidate_uuid: "5dc64e9212cb45f98900448483df54ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b61b2310370b4c878bca741b0ca025f7"
I20260927 18:50:51.759042 17844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7cad36dc654279b5f063fc7fdbe559" candidate_uuid: "5dc64e9212cb45f98900448483df54ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b1518365134184bbf0ec6eb73479a2"
I20260927 18:50:51.759071 17713 raft_consensus.cc:3037] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:51.759186 17844 raft_consensus.cc:3037] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:51.760215 17713 raft_consensus.cc:2454] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dc64e9212cb45f98900448483df54ed in term 1.
I20260927 18:50:51.760217 17844 raft_consensus.cc:2454] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dc64e9212cb45f98900448483df54ed in term 1.
I20260927 18:50:51.760555 17909 leader_election.cc:302] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc64e9212cb45f98900448483df54ed, c1b1518365134184bbf0ec6eb73479a2; no voters:
I20260927 18:50:51.760737 18046 raft_consensus.cc:2781] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:51.760931 18046 raft_consensus.cc:686] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 1 LEADER]: Becoming Leader. State: Replica: 5dc64e9212cb45f98900448483df54ed, State: Running, Role: LEADER
I20260927 18:50:51.761255 18046 consensus_queue.cc:237] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc64e9212cb45f98900448483df54ed" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45563 } } peers { permanent_uuid: "b61b2310370b4c878bca741b0ca025f7" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46071 } } peers { permanent_uuid: "c1b1518365134184bbf0ec6eb73479a2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45459 } }
I20260927 18:50:51.764508 17570 catalog_manager.cc:5495] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dc64e9212cb45f98900448483df54ed (127.29.159.67). New cstate: current_term: 1 leader_uuid: "5dc64e9212cb45f98900448483df54ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc64e9212cb45f98900448483df54ed" member_type: VOTER last_known_addr { host: "127.29.159.67" port: 45563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b61b2310370b4c878bca741b0ca025f7" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1b1518365134184bbf0ec6eb73479a2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45459 } health_report { overall_health: UNKNOWN } } }
W20260927 18:50:51.770025 18022 tablet.cc:2369] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:50:51.807363 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.803585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.29.159.67:45563)
0927 18:50:51.803750 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:51.803756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:51.803772 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:50:51.803957 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:51.803960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:51.804026 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:51.804139 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:50:51.804389 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.804892 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.805792 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:51.806038 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:51.806148 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:51.806478 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:50:51.806913 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:50:51.807064 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:51.807128 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 18:50:51.807183 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:50:51.813714 17844 raft_consensus.cc:1261] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2 [term 1 FOLLOWER]: Refusing update from remote peer 5dc64e9212cb45f98900448483df54ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:51.813764 17713 raft_consensus.cc:1261] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7 [term 1 FOLLOWER]: Refusing update from remote peer 5dc64e9212cb45f98900448483df54ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:50:51.814493 18048 consensus_queue.cc:1035] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b1518365134184bbf0ec6eb73479a2" member_type: VOTER last_known_addr { host: "127.29.159.66" port: 45459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:50:51.814754 18046 consensus_queue.cc:1035] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61b2310370b4c878bca741b0ca025f7" member_type: VOTER last_known_addr { host: "127.29.159.65" port: 46071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:50:51.820942 17891 tablet.cc:2369] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:50:51.822742 18053 mvcc.cc:205] Tried to move back new op lower bound from 7334031572221124608 to 7334031572016799744. Current Snapshot: MvccSnapshot[applied={T|T < 7334031572221124608}]
I20260927 18:50:51.830775 17571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:51.830964 17571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:50:51.833775 17571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5f811c20383484c8ba5f7e39ad2668d: Sending DeleteTablet for 3 replicas of tablet cc7cad36dc654279b5f063fc7fdbe559
I20260927 18:50:51.834642 17955 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7cad36dc654279b5f063fc7fdbe559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:51 UTC) from {username='slave'} at 127.0.0.1:57224
I20260927 18:50:51.834676 17693 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7cad36dc654279b5f063fc7fdbe559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:51 UTC) from {username='slave'} at 127.0.0.1:33316
I20260927 18:50:51.835196 18060 tablet_replica.cc:330] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: stopping tablet replica
I20260927 18:50:51.835225 18061 tablet_replica.cc:330] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7: stopping tablet replica
I20260927 18:50:51.835242 17823 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7cad36dc654279b5f063fc7fdbe559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:50:51 UTC) from {username='slave'} at 127.0.0.1:59478
I20260927 18:50:51.835601 18061 raft_consensus.cc:2229] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:51.835718 18060 raft_consensus.cc:2229] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:50:51.835981 18061 raft_consensus.cc:2258] T cc7cad36dc654279b5f063fc7fdbe559 P b61b2310370b4c878bca741b0ca025f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:51.836230 18060 raft_consensus.cc:2258] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:51.836652 18062 tablet_replica.cc:330] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: stopping tablet replica
I20260927 18:50:51.836736 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 17631
I20260927 18:50:51.837925 18062 raft_consensus.cc:2229] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:50:51.838295 18062 raft_consensus.cc:2258] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:50:51.838418 18060 ts_tablet_manager.cc:1905] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:51.840065 18062 ts_tablet_manager.cc:1905] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:50:51.841573 18060 ts_tablet_manager.cc:1918] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:51.841683 18060 log.cc:1196] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-2/wal/wals/cc7cad36dc654279b5f063fc7fdbe559
I20260927 18:50:51.842061 18060 ts_tablet_manager.cc:1939] T cc7cad36dc654279b5f063fc7fdbe559 P 5dc64e9212cb45f98900448483df54ed: Deleting consensus metadata
I20260927 18:50:51.843333 17558 catalog_manager.cc:4841] TS 5dc64e9212cb45f98900448483df54ed (127.29.159.67:45563): tablet cc7cad36dc654279b5f063fc7fdbe559 (table test-table [id=9013215d9ca6464c862ef7dd07b13020]) successfully deleted
I20260927 18:50:51.843789 18062 ts_tablet_manager.cc:1918] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:50:51.843968 18062 log.cc:1196] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: Deleting WAL directory at /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534801142164-30333-0/minicluster-data/ts-1/wal/wals/cc7cad36dc654279b5f063fc7fdbe559
I20260927 18:50:51.844357 18062 ts_tablet_manager.cc:1939] T cc7cad36dc654279b5f063fc7fdbe559 P c1b1518365134184bbf0ec6eb73479a2: Deleting consensus metadata
I20260927 18:50:51.845647 17556 catalog_manager.cc:4841] TS c1b1518365134184bbf0ec6eb73479a2 (127.29.159.66:45459): tablet cc7cad36dc654279b5f063fc7fdbe559 (table test-table [id=9013215d9ca6464c862ef7dd07b13020]) successfully deleted
W20260927 18:50:51.845969 17557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.159.65:46071 (error 108)
I20260927 18:50:51.846176 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 17762
I20260927 18:50:51.847651 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:51.847251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.65:46071 (local address 127.0.0.1:33324)
0927 18:50:51.847465 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:50:51.847545 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.65:46071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260927 18:50:51.847931 17557 catalog_manager.cc:4584] TS b61b2310370b4c878bca741b0ca025f7 (127.29.159.65:46071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc7cad36dc654279b5f063fc7fdbe559: Network error: Client connection negotiation failed: client connection to 127.29.159.65:46071: connect: Connection refused (error 111)
I20260927 18:50:51.852242 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 17893
I20260927 18:50:51.858572 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 17540
2026-09-27T18:50:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1224 ms)
[----------] 6 tests from EncryptionPolicyTest (10324 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:51 dist-test-slave-8tg2 krb5kdc[18066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:51 dist-test-slave-8tg2 krb5kdc[18066](info): set up 2 sockets
Sep 27 18:50:51 dist-test-slave-8tg2 krb5kdc[18066](info): commencing operation
krb5kdc: starting...
W20260927 18:50:52.919692 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:50:53 dist-test-slave-8tg2 krb5kdc[18066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:53.046799 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:53.155974 18079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:53.156220 18079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:53.156260 18079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:53.160354 18079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:53.160418 18079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:53.160444 18079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:53.160462 18079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:53.160477 18079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:53.165812 18079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39421
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:53.166743 18079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:53.167698 18079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:53.175866 18087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:53.175846 18084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:53.175868 18085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:53.176014 18079 server_base.cc:1034] running on GCE node
I20260927 18:50:53.176769 18079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:53.177832 18079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:53.179019 18079 hybrid_clock.cc:648] HybridClock initialized: now 1790535053179010 us; error 39 us; skew 500 ppm
Sep 27 18:50:53 dist-test-slave-8tg2 krb5kdc[18066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:53.181702 18079 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:53.182660 18079 webserver.cc:460] Webserver started at http://127.0.0.1:39409/ using document root <none> and password file <none>
I20260927 18:50:53.183205 18079 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:53.183293 18079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:53.183584 18079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:53.185588 18079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "ef1737049833419cbaaa4c2a3d605942"
format_stamp: "Formatted at 2026-09-27 18:50:53 on dist-test-slave-8tg2"
server_key: "b62209ac4ec4e1f2408947e0f7af2f50"
server_key_iv: "a1d62ea99418e3944a0fafb208dca271"
server_key_version: "encryptionkey@0"
I20260927 18:50:53.186134 18079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "ef1737049833419cbaaa4c2a3d605942"
format_stamp: "Formatted at 2026-09-27 18:50:53 on dist-test-slave-8tg2"
server_key: "b62209ac4ec4e1f2408947e0f7af2f50"
server_key_iv: "a1d62ea99418e3944a0fafb208dca271"
server_key_version: "encryptionkey@0"
I20260927 18:50:53.189719 18079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:50:53.192055 18094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:53.193217 18079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:50:53.193351 18079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "ef1737049833419cbaaa4c2a3d605942"
format_stamp: "Formatted at 2026-09-27 18:50:53 on dist-test-slave-8tg2"
server_key: "b62209ac4ec4e1f2408947e0f7af2f50"
server_key_iv: "a1d62ea99418e3944a0fafb208dca271"
server_key_version: "encryptionkey@0"
I20260927 18:50:53.193466 18079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:53.211808 18079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:53.214982 18079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:53.215183 18079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:53.222838 18079 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39421
I20260927 18:50:53.222957 18146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39421 every 8 connection(s)
I20260927 18:50:53.223912 18079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:53.226747 18147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:53.232246 18147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942: Bootstrap starting.
I20260927 18:50:53.233068 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 18079
I20260927 18:50:53.233204 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:50:53.233480 30333 external_mini_cluster.cc:1428] Setting key 9c08238664eecbd86aa36dcadd85057a
I20260927 18:50:53.234899 18147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:53.235688 18147 log.cc:824] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:53.237454 18147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942: No bootstrap required, opened a new log
I20260927 18:50:53.240964 18147 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1737049833419cbaaa4c2a3d605942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39421 } }
I20260927 18:50:53.241173 18147 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:53.241230 18147 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef1737049833419cbaaa4c2a3d605942, State: Initialized, Role: FOLLOWER
I20260927 18:50:53.241709 18147 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1737049833419cbaaa4c2a3d605942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39421 } }
I20260927 18:50:53.241891 18147 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:53.241955 18147 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:53.242060 18147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 18:50:53 dist-test-slave-8tg2 krb5kdc[18066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:50:53.243189 18147 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1737049833419cbaaa4c2a3d605942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39421 } }
I20260927 18:50:53.243582 18147 leader_election.cc:302] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef1737049833419cbaaa4c2a3d605942; no voters:
I20260927 18:50:53.243956 18147 leader_election.cc:288] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:53.244097 18152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:53.244323 18152 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [term 1 LEADER]: Becoming Leader. State: Replica: ef1737049833419cbaaa4c2a3d605942, State: Running, Role: LEADER
I20260927 18:50:53.244647 18152 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1737049833419cbaaa4c2a3d605942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39421 } }
I20260927 18:50:53.245190 18147 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:53.246047 18153 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef1737049833419cbaaa4c2a3d605942. Latest consensus state: current_term: 1 leader_uuid: "ef1737049833419cbaaa4c2a3d605942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1737049833419cbaaa4c2a3d605942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39421 } } }
I20260927 18:50:53.246192 18153 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:53.246425 18154 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef1737049833419cbaaa4c2a3d605942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1737049833419cbaaa4c2a3d605942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39421 } } }
I20260927 18:50:53.246605 18154 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:53.246590 18161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:53.248741 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:53.234591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36153 (local address 127.0.0.1:39421)
0927 18:50:53.235053 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:53.235070 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:53.235104 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:50:53.235855 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:53.235873 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:53.236205 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:53.236527 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:53.237343 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:53.238359 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:53.239245 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:53.239814 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:53.239997 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:53.242994 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:53.242999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:53.243012 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:53.243037 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:53.245113 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:53.245744 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:53.245750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:53.245756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:53.245881 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:53.246311 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:53.246316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:53.246318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:53.246838 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:53.247053 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:53.247559 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 18:50:53.247786 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":148,"threads_started":1}
I20260927 18:50:53.249929 18161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:53.253242 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:53.263998 18161 catalog_manager.cc:1347] Generated new cluster ID: d6ca8a0328dc4c5c8791abddd51da4c8
I20260927 18:50:53.264066 18161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:53.275943 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:53.255810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57065 (local address 127.0.0.1:39421)
0927 18:50:53.255921 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:53.255926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:53.255946 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:50:53.256035 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:53.256037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:53.256071 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:53.256135 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:53.256932 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:53.257469 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:53.258597 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:53.258832 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:53.258907 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:53.260539 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:53.260542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:53.260544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:53.260565 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:53.262217 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:53.262704 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:53.262708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:53.262710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:53.262788 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:53.263124 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:53.263127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:53.263129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:53.263295 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:53.263508 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:53.263763 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 18:50:53.263855 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:50:53.285471 18161 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:53.286751 18161 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:53.297694 18161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef1737049833419cbaaa4c2a3d605942: Generated new TSK 0
I20260927 18:50:53.298502 18161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:53.303834 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 18079
W20260927 18:50:53.308806 18064 connection.cc:535] client connection to 127.0.0.1:39421 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39421): Connection reset by peer (error 104)
2026-09-27T18:50:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:53 dist-test-slave-8tg2 krb5kdc[18178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:53 dist-test-slave-8tg2 krb5kdc[18178](info): set up 2 sockets
Sep 27 18:50:53 dist-test-slave-8tg2 krb5kdc[18178](info): commencing operation
krb5kdc: starting...
W20260927 18:50:54.339691 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:50:54 dist-test-slave-8tg2 krb5kdc[18178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:54.466856 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:54.581794 18191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:54.582067 18191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:54.582109 18191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:54.586104 18191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:54.586165 18191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:54.586185 18191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:54.586202 18191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:54.586217 18191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:54.591786 18191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:54.592939 18191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:54.593932 18191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:54.602823 18196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:54.602873 18199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:54.603039 18191 server_base.cc:1034] running on GCE node
W20260927 18:50:54.602819 18197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:54.603942 18191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:54.605119 18191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:54.606287 18191 hybrid_clock.cc:648] HybridClock initialized: now 1790535054606270 us; error 33 us; skew 500 ppm
Sep 27 18:50:54 dist-test-slave-8tg2 krb5kdc[18178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:54.609157 18191 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:54.610086 18191 webserver.cc:460] Webserver started at http://127.0.0.1:38191/ using document root <none> and password file <none>
I20260927 18:50:54.610648 18191 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:54.610741 18191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:54.610981 18191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:54.613170 18191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "f8243d08230d48d1a53c0848764475e4"
format_stamp: "Formatted at 2026-09-27 18:50:54 on dist-test-slave-8tg2"
server_key: "00da36de7a6ac6b8debe940163899f56"
server_key_iv: "a06c4d0b0df6b6a60fa1bb4dcb17ef9c"
server_key_version: "encryptionkey@0"
I20260927 18:50:54.613737 18191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "f8243d08230d48d1a53c0848764475e4"
format_stamp: "Formatted at 2026-09-27 18:50:54 on dist-test-slave-8tg2"
server_key: "00da36de7a6ac6b8debe940163899f56"
server_key_iv: "a06c4d0b0df6b6a60fa1bb4dcb17ef9c"
server_key_version: "encryptionkey@0"
I20260927 18:50:54.617492 18191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260927 18:50:54.619987 18206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:54.621276 18191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:54.621418 18191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "f8243d08230d48d1a53c0848764475e4"
format_stamp: "Formatted at 2026-09-27 18:50:54 on dist-test-slave-8tg2"
server_key: "00da36de7a6ac6b8debe940163899f56"
server_key_iv: "a06c4d0b0df6b6a60fa1bb4dcb17ef9c"
server_key_version: "encryptionkey@0"
I20260927 18:50:54.621533 18191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:54.634953 18191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:54.638056 18191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:54.638257 18191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:54.645393 18191 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36815
I20260927 18:50:54.645409 18258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36815 every 8 connection(s)
I20260927 18:50:54.646351 18191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:54.649156 18259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:54.653844 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 18191
I20260927 18:50:54.654078 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:50:54.654352 30333 external_mini_cluster.cc:1428] Setting key 2af01cf45040ec92f494be2b49a3b57c
I20260927 18:50:54.654671 18259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4: Bootstrap starting.
I20260927 18:50:54.657331 18259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:54.658175 18259 log.cc:824] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:54.660234 18259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4: No bootstrap required, opened a new log
I20260927 18:50:54.664301 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:54.655775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37279 (local address 127.0.0.1:36815)
0927 18:50:54.656181 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:54.656199 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:54.656240 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:50:54.657022 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:54.657044 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:54.657475 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:54.657844 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:54.658271 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:54.659651 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:54.660574 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:54.661232 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:54.661398 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:54.662195 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:54.662202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:54.662218 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:54.662699 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:54.662770 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:54.662799 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:54.662964 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:54.663168 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 18:50:54.663327 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":108,"threads_started":1}
I20260927 18:50:54.664707 18259 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8243d08230d48d1a53c0848764475e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36815 } }
I20260927 18:50:54.664965 18259 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:54.665027 18259 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8243d08230d48d1a53c0848764475e4, State: Initialized, Role: FOLLOWER
I20260927 18:50:54.665469 18259 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8243d08230d48d1a53c0848764475e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36815 } }
I20260927 18:50:54.665598 18259 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:54.665639 18259 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:54.665783 18259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:54.666846 18259 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8243d08230d48d1a53c0848764475e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36815 } }
I20260927 18:50:54.667200 18259 leader_election.cc:302] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8243d08230d48d1a53c0848764475e4; no voters:
I20260927 18:50:54.667502 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:54.667613 18259 leader_election.cc:288] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:54.667676 18264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:54.667881 18264 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [term 1 LEADER]: Becoming Leader. State: Replica: f8243d08230d48d1a53c0848764475e4, State: Running, Role: LEADER
I20260927 18:50:54.668174 18264 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8243d08230d48d1a53c0848764475e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36815 } }
I20260927 18:50:54.668900 18259 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:54.669318 18264 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8243d08230d48d1a53c0848764475e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8243d08230d48d1a53c0848764475e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36815 } } }
I20260927 18:50:54.669425 18264 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:54.669730 18265 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8243d08230d48d1a53c0848764475e4. Latest consensus state: current_term: 1 leader_uuid: "f8243d08230d48d1a53c0848764475e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8243d08230d48d1a53c0848764475e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36815 } } }
I20260927 18:50:54.669806 18272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:54.669824 18265 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:54.673306 18272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:54.674997 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:54.670517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59669 (local address 127.0.0.1:36815)
0927 18:50:54.670609 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:54.670613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:54.670653 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:50:54.670688 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:54.670691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:54.670733 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:54.670827 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:54.671180 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:54.672013 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:54.673180 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:54.673526 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:54.673631 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:54.674510 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:54.674514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:54.674517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:54.674529 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:54.674557 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:54.674570 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:54.674658 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:54.674828 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 18:50:54.674863 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 18:50:54.679091 18272 catalog_manager.cc:1347] Generated new cluster ID: 542dd8c2c9f947aea378f70f4216d0af
I20260927 18:50:54.679164 18272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:54.700470 18272 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:54.701421 18272 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:54.713999 18272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8243d08230d48d1a53c0848764475e4: Generated new TSK 0
I20260927 18:50:54.714775 18272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:54.721464 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 18191
W20260927 18:50:54.727581 18176 connection.cc:535] client connection to 127.0.0.1:36815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36815): Connection reset by peer (error 104)
2026-09-27T18:50:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:54 dist-test-slave-8tg2 krb5kdc[18290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:54 dist-test-slave-8tg2 krb5kdc[18290](info): set up 2 sockets
Sep 27 18:50:54 dist-test-slave-8tg2 krb5kdc[18290](info): commencing operation
krb5kdc: starting...
W20260927 18:50:55.775701 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:50:55 dist-test-slave-8tg2 krb5kdc[18290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:55.894490 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:56.006606 18303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:56.006930 18303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:56.006975 18303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:56.011350 18303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:56.011446 18303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:56.011468 18303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:56.011487 18303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:56.011543 18303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:56.017638 18303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40773
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:56.018920 18303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:56.020080 18303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:56.029441 18308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:56.029520 18309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:56.029666 18303 server_base.cc:1034] running on GCE node
W20260927 18:50:56.029844 18311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:56.030414 18303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:56.031761 18303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:56.032938 18303 hybrid_clock.cc:648] HybridClock initialized: now 1790535056032919 us; error 33 us; skew 500 ppm
Sep 27 18:50:56 dist-test-slave-8tg2 krb5kdc[18290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:56.035758 18303 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:56.036698 18303 webserver.cc:460] Webserver started at http://127.0.0.1:41433/ using document root <none> and password file <none>
I20260927 18:50:56.037297 18303 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:56.037364 18303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:56.037586 18303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:56.039817 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "c152d54601bf46f7bd79b8a4edc15604"
format_stamp: "Formatted at 2026-09-27 18:50:56 on dist-test-slave-8tg2"
server_key: "3c34c07a002c12cf43b7bb9cec4e6d41"
server_key_iv: "f522fb2beaef6b027644daa806a29301"
server_key_version: "encryptionkey@0"
I20260927 18:50:56.040398 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "c152d54601bf46f7bd79b8a4edc15604"
format_stamp: "Formatted at 2026-09-27 18:50:56 on dist-test-slave-8tg2"
server_key: "3c34c07a002c12cf43b7bb9cec4e6d41"
server_key_iv: "f522fb2beaef6b027644daa806a29301"
server_key_version: "encryptionkey@0"
I20260927 18:50:56.044344 18303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:50:56.046917 18318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:56.048373 18303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:50:56.048537 18303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "c152d54601bf46f7bd79b8a4edc15604"
format_stamp: "Formatted at 2026-09-27 18:50:56 on dist-test-slave-8tg2"
server_key: "3c34c07a002c12cf43b7bb9cec4e6d41"
server_key_iv: "f522fb2beaef6b027644daa806a29301"
server_key_version: "encryptionkey@0"
I20260927 18:50:56.048659 18303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:56.053589 18303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:56.056797 18303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:56.057013 18303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:56.064226 18303 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40773
I20260927 18:50:56.064278 18370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40773 every 8 connection(s)
I20260927 18:50:56.065245 18303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:56.068156 18371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:56.071367 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 18303
I20260927 18:50:56.071542 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:50:56.071880 30333 external_mini_cluster.cc:1428] Setting key 161eea502a0638e5699d91b6c664476b
I20260927 18:50:56.074496 18371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604: Bootstrap starting.
I20260927 18:50:56.077067 18371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:56.077582 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:56.073438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52999 (local address 127.0.0.1:40773)
0927 18:50:56.073910 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:56.073926 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:56.073964 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:50:56.074710 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:56.074730 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:56.075038 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:56.075390 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:56.076311 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:56.076316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:56.076328 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:56.076338 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:56.076392 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:56.076411 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:56.076559 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:56.076635 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 18:50:56.076805 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":150,"threads_started":1}
I20260927 18:50:56.077940 18371 log.cc:824] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:56.080070 18371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604: No bootstrap required, opened a new log
I20260927 18:50:56.080605 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:56.084646 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:56.082694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59869 (local address 127.0.0.1:40773)
0927 18:50:56.082849 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:56.082854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:56.083164 (+ 310us) server_negotiation.cc:407] Connection header received
0927 18:50:56.083288 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:56.083290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:56.083322 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:56.083388 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:56.084164 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:56.084167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:56.084169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:56.084179 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:56.084203 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:56.084213 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:56.084283 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:56.084395 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 18:50:56.084437 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 18:50:56.084520 18371 raft_consensus.cc:348] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c152d54601bf46f7bd79b8a4edc15604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260927 18:50:56.084853 18371 raft_consensus.cc:374] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:56.084913 18371 raft_consensus.cc:729] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c152d54601bf46f7bd79b8a4edc15604, State: Initialized, Role: FOLLOWER
I20260927 18:50:56.085405 18371 consensus_queue.cc:260] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c152d54601bf46f7bd79b8a4edc15604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260927 18:50:56.085590 18371 raft_consensus.cc:388] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:56.085667 18371 raft_consensus.cc:482] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:56.085772 18371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:56.086760 18371 raft_consensus.cc:504] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c152d54601bf46f7bd79b8a4edc15604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260927 18:50:56.087091 18371 leader_election.cc:302] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c152d54601bf46f7bd79b8a4edc15604; no voters:
I20260927 18:50:56.087354 18371 leader_election.cc:288] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:56.087512 18381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:56.087736 18381 raft_consensus.cc:686] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [term 1 LEADER]: Becoming Leader. State: Replica: c152d54601bf46f7bd79b8a4edc15604, State: Running, Role: LEADER
I20260927 18:50:56.088114 18381 consensus_queue.cc:237] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c152d54601bf46f7bd79b8a4edc15604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260927 18:50:56.088814 18371 sys_catalog.cc:564] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:56.089916 18382 sys_catalog.cc:455] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c152d54601bf46f7bd79b8a4edc15604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c152d54601bf46f7bd79b8a4edc15604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } } }
I20260927 18:50:56.090057 18382 sys_catalog.cc:458] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:56.090376 18383 sys_catalog.cc:455] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c152d54601bf46f7bd79b8a4edc15604. Latest consensus state: current_term: 1 leader_uuid: "c152d54601bf46f7bd79b8a4edc15604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c152d54601bf46f7bd79b8a4edc15604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } } }
I20260927 18:50:56.090482 18390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:56.090565 18383 sys_catalog.cc:458] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:56.093868 18390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:56.100266 18390 catalog_manager.cc:1347] Generated new cluster ID: 90eebfa26b6d4ea2b00908bbc27d4346
I20260927 18:50:56.100353 18390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:56.121351 18390 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:56.121784 18390 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:56.129340 18390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c152d54601bf46f7bd79b8a4edc15604: Generated new TSK 0
I20260927 18:50:56.130309 18390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:56.140817 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 18303
2026-09-27T18:50:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:56 dist-test-slave-8tg2 krb5kdc[18402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:56 dist-test-slave-8tg2 krb5kdc[18402](info): set up 2 sockets
Sep 27 18:50:56 dist-test-slave-8tg2 krb5kdc[18402](info): commencing operation
krb5kdc: starting...
W20260927 18:50:57.179812 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:50:57 dist-test-slave-8tg2 krb5kdc[18402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:57.304567 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:57.418565 18415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:57.418883 18415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:57.418972 18415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:57.423209 18415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:57.423316 18415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:57.423355 18415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:57.423396 18415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:57.423493 18415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:57.429021 18415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:57.430235 18415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:57.431274 18415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:57.439733 18423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:57.439736 18420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:57.439772 18421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:57.440116 18415 server_base.cc:1034] running on GCE node
I20260927 18:50:57.440653 18415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:57.441740 18415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:57.442955 18415 hybrid_clock.cc:648] HybridClock initialized: now 1790535057442927 us; error 40 us; skew 500 ppm
Sep 27 18:50:57 dist-test-slave-8tg2 krb5kdc[18402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:57.454201 18415 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:57.455267 18415 webserver.cc:460] Webserver started at http://127.0.0.1:40713/ using document root <none> and password file <none>
I20260927 18:50:57.455962 18415 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:57.456064 18415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:57.456306 18415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:57.458452 18415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "23aa9b4f23ef4e7a9f5a9e1f5f43a081"
format_stamp: "Formatted at 2026-09-27 18:50:57 on dist-test-slave-8tg2"
server_key: "1b1f531eddce474df3da945021b56bcc"
server_key_iv: "08bc6db1a291f3bf2dd130cac2b62a28"
server_key_version: "encryptionkey@0"
I20260927 18:50:57.459045 18415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "23aa9b4f23ef4e7a9f5a9e1f5f43a081"
format_stamp: "Formatted at 2026-09-27 18:50:57 on dist-test-slave-8tg2"
server_key: "1b1f531eddce474df3da945021b56bcc"
server_key_iv: "08bc6db1a291f3bf2dd130cac2b62a28"
server_key_version: "encryptionkey@0"
I20260927 18:50:57.463133 18415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 18:50:57.465754 18430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:57.467130 18415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:50:57.467274 18415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "23aa9b4f23ef4e7a9f5a9e1f5f43a081"
format_stamp: "Formatted at 2026-09-27 18:50:57 on dist-test-slave-8tg2"
server_key: "1b1f531eddce474df3da945021b56bcc"
server_key_iv: "08bc6db1a291f3bf2dd130cac2b62a28"
server_key_version: "encryptionkey@0"
I20260927 18:50:57.467396 18415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:57.498579 18415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:57.502153 18415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:57.502372 18415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:57.509938 18415 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35641
I20260927 18:50:57.509963 18482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35641 every 8 connection(s)
I20260927 18:50:57.510969 18415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:57.511658 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 18415
I20260927 18:50:57.511837 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:50:57.512142 30333 external_mini_cluster.cc:1428] Setting key 31357934f7e46d67d9f0be7a0b9f41e6
I20260927 18:50:57.514493 18483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:57.521488 18483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081: Bootstrap starting.
Sep 27 18:50:57 dist-test-slave-8tg2 krb5kdc[18402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:50:57.524556 18483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:57.525506 18483 log.cc:824] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:57.527706 18483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081: No bootstrap required, opened a new log
I20260927 18:50:57.528389 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:57.513722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50681 (local address 127.0.0.1:35641)
0927 18:50:57.514155 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:57.514178 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:57.514224 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:50:57.515077 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:57.515101 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:57.515587 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:57.515959 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:57.516731 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:57.518228 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:57.519140 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:57.519894 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:57.520069 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:57.522945 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:57.522953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:57.522973 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:57.523008 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:57.525191 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:57.525718 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:57.525727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:57.525739 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:57.525875 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:57.526198 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:57.526202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:57.526204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:57.526691 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:57.526905 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:57.527240 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 18:50:57.527591 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260927 18:50:57.531685 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:57.532079 18483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aa9b4f23ef4e7a9f5a9e1f5f43a081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260927 18:50:57.532339 18483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:57.532428 18483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23aa9b4f23ef4e7a9f5a9e1f5f43a081, State: Initialized, Role: FOLLOWER
I20260927 18:50:57.532893 18483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aa9b4f23ef4e7a9f5a9e1f5f43a081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260927 18:50:57.533078 18483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:57.533150 18483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:57.533278 18483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:57.534729 18483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aa9b4f23ef4e7a9f5a9e1f5f43a081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260927 18:50:57.535161 18483 leader_election.cc:302] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23aa9b4f23ef4e7a9f5a9e1f5f43a081; no voters:
I20260927 18:50:57.535566 18483 leader_election.cc:288] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:57.535691 18493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:57.535951 18493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [term 1 LEADER]: Becoming Leader. State: Replica: 23aa9b4f23ef4e7a9f5a9e1f5f43a081, State: Running, Role: LEADER
I20260927 18:50:57.536391 18493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aa9b4f23ef4e7a9f5a9e1f5f43a081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260927 18:50:57.536940 18483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:57.537571 18493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23aa9b4f23ef4e7a9f5a9e1f5f43a081. Latest consensus state: current_term: 1 leader_uuid: "23aa9b4f23ef4e7a9f5a9e1f5f43a081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aa9b4f23ef4e7a9f5a9e1f5f43a081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } } }
I20260927 18:50:57.537756 18493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:57.538162 18494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23aa9b4f23ef4e7a9f5a9e1f5f43a081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aa9b4f23ef4e7a9f5a9e1f5f43a081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } } }
I20260927 18:50:57.538326 18494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081 [sys.catalog]: This master's current role is: LEADER
W20260927 18:50:57.542730 18508 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:50:57.542810 18508 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:50:57.542936 18501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:57.543491 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:57.533839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59761 (local address 127.0.0.1:35641)
0927 18:50:57.534051 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:57.534058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:57.534262 (+ 204us) server_negotiation.cc:407] Connection header received
0927 18:50:57.534435 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:57.534437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:57.534480 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:57.534557 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:57.535407 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:57.536257 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:57.537395 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:57.537774 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:57.537887 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:57.539776 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:57.539779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:57.539782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:50:57.539806 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:57.541525 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:57.542011 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:57.542015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:57.542017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:57.542103 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:50:57.542418 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:50:57.542421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:50:57.542423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:50:57.542641 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:57.542773 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:57.543054 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 18:50:57.543173 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:50:57.543838 18501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:57.549687 18501 catalog_manager.cc:1347] Generated new cluster ID: 646abcd7fcbe4db198e32bdb242f4bca
I20260927 18:50:57.549779 18501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:57.554551 18501 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:57.555513 18501 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:57.561296 18501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23aa9b4f23ef4e7a9f5a9e1f5f43a081: Generated new TSK 0
I20260927 18:50:57.561991 18501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:57.569749 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 18415
2026-09-27T18:50:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:57 dist-test-slave-8tg2 krb5kdc[18514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:57 dist-test-slave-8tg2 krb5kdc[18514](info): set up 2 sockets
Sep 27 18:50:57 dist-test-slave-8tg2 krb5kdc[18514](info): commencing operation
krb5kdc: starting...
W20260927 18:50:58.607764 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:50:58 dist-test-slave-8tg2 krb5kdc[18514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:50:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:50:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:50:58.731154 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:50:58.842355 18527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:50:58.842705 18527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:50:58.842785 18527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:50:58.846802 18527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:50:58.846892 18527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:50:58.846930 18527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:50:58.846987 18527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:50:58.847043 18527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:50:58.852377 18527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:50:58.853398 18527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:50:58.854462 18527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:50:58.862823 18535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:58.862884 18533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:50:58.862886 18532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:50:58.862938 18527 server_base.cc:1034] running on GCE node
I20260927 18:50:58.863662 18527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:50:58.864705 18527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:50:58.865891 18527 hybrid_clock.cc:648] HybridClock initialized: now 1790535058865871 us; error 37 us; skew 500 ppm
Sep 27 18:50:58 dist-test-slave-8tg2 krb5kdc[18514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:50:58.868688 18527 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:50:58.869652 18527 webserver.cc:460] Webserver started at http://127.0.0.1:36287/ using document root <none> and password file <none>
I20260927 18:50:58.870200 18527 fs_manager.cc:359] Metadata directory not provided
I20260927 18:50:58.870299 18527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:50:58.870537 18527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:50:58.872625 18527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "aac953d79124496f95359328001b0454"
format_stamp: "Formatted at 2026-09-27 18:50:58 on dist-test-slave-8tg2"
server_key: "af90ade7cf63c2ad5c4070fece415b90"
server_key_iv: "927567655b78aba248cd866b701e218a"
server_key_version: "encryptionkey@0"
I20260927 18:50:58.873183 18527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "aac953d79124496f95359328001b0454"
format_stamp: "Formatted at 2026-09-27 18:50:58 on dist-test-slave-8tg2"
server_key: "af90ade7cf63c2ad5c4070fece415b90"
server_key_iv: "927567655b78aba248cd866b701e218a"
server_key_version: "encryptionkey@0"
I20260927 18:50:58.876765 18527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:50:58.879012 18542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:50:58.880234 18527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:50:58.880373 18527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "aac953d79124496f95359328001b0454"
format_stamp: "Formatted at 2026-09-27 18:50:58 on dist-test-slave-8tg2"
server_key: "af90ade7cf63c2ad5c4070fece415b90"
server_key_iv: "927567655b78aba248cd866b701e218a"
server_key_version: "encryptionkey@0"
I20260927 18:50:58.880498 18527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:50:58.914156 18527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:50:58.917701 18527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:50:58.917925 18527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:50:58.925144 18527 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38727
I20260927 18:50:58.925168 18594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38727 every 8 connection(s)
I20260927 18:50:58.926107 18527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:50:58.927742 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 18527
I20260927 18:50:58.927903 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:50:58.928213 30333 external_mini_cluster.cc:1428] Setting key 85ba87cde549e887766a5ad4e46b71ba
I20260927 18:50:58.929260 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:50:58.934724 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454: Bootstrap starting.
I20260927 18:50:58.937057 18595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454: Neither blocks nor log segments found. Creating new log.
I20260927 18:50:58.937167 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:58.929633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51129 (local address 127.0.0.1:38727)
0927 18:50:58.930140 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:58.930165 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:58.930199 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:50:58.930827 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:58.930843 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:58.931183 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:58.931540 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:58.931937 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:58.932903 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:58.933809 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:58.934285 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:58.934459 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:58.935365 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:58.935371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:58.935383 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:58.935832 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:58.935884 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:58.935907 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:58.936027 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:58.936154 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 18:50:58.936368 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":128,"threads_started":1}
I20260927 18:50:58.937929 18595 log.cc:824] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454: Log is configured to *not* fsync() on all Append() calls
I20260927 18:50:58.939951 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:50:58.940086 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454: No bootstrap required, opened a new log
I20260927 18:50:58.944080 18595 raft_consensus.cc:348] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac953d79124496f95359328001b0454" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260927 18:50:58.944347 18595 raft_consensus.cc:374] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:50:58.944415 18595 raft_consensus.cc:729] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac953d79124496f95359328001b0454, State: Initialized, Role: FOLLOWER
I20260927 18:50:58.944842 18595 consensus_queue.cc:260] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac953d79124496f95359328001b0454" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260927 18:50:58.945005 18595 raft_consensus.cc:388] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:50:58.945084 18595 raft_consensus.cc:482] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:50:58.945195 18595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:50:58.946075 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:50:58.941893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36233 (local address 127.0.0.1:38727)
0927 18:50:58.942019 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:50:58.942023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:50:58.942207 (+ 184us) server_negotiation.cc:407] Connection header received
0927 18:50:58.942347 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:50:58.942349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:50:58.942383 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:50:58.942453 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:50:58.942806 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:58.943287 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:58.944377 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:50:58.944752 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:50:58.944834 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:50:58.945679 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:50:58.945681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:50:58.945683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:50:58.945694 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:50:58.945716 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:50:58.945726 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:50:58.945785 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 18:50:58.945894 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 18:50:58.945935 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:50:58.946363 18595 raft_consensus.cc:504] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac953d79124496f95359328001b0454" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260927 18:50:58.946715 18595 leader_election.cc:302] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aac953d79124496f95359328001b0454; no voters:
I20260927 18:50:58.947029 18595 leader_election.cc:288] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:50:58.947093 18605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:50:58.947309 18605 raft_consensus.cc:686] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [term 1 LEADER]: Becoming Leader. State: Replica: aac953d79124496f95359328001b0454, State: Running, Role: LEADER
I20260927 18:50:58.947669 18605 consensus_queue.cc:237] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac953d79124496f95359328001b0454" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } }
I20260927 18:50:58.948308 18595 sys_catalog.cc:564] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:50:58.949302 18607 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aac953d79124496f95359328001b0454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac953d79124496f95359328001b0454" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } } }
I20260927 18:50:58.949430 18607 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:58.949815 18614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:50:58.950443 18606 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac953d79124496f95359328001b0454. Latest consensus state: current_term: 1 leader_uuid: "aac953d79124496f95359328001b0454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac953d79124496f95359328001b0454" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38727 } } }
I20260927 18:50:58.950589 18606 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454 [sys.catalog]: This master's current role is: LEADER
I20260927 18:50:58.952561 18614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:50:58.958002 18614 catalog_manager.cc:1347] Generated new cluster ID: af1decb76763495fad7d430623561ba4
I20260927 18:50:58.958062 18614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:50:58.994390 18614 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:50:58.995396 18614 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:50:59.001767 18614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aac953d79124496f95359328001b0454: Generated new TSK 0
I20260927 18:50:59.002475 18614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:50:59.008980 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 18527
2026-09-27T18:50:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:50:59 dist-test-slave-8tg2 krb5kdc[18626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:50:59 dist-test-slave-8tg2 krb5kdc[18626](info): set up 2 sockets
Sep 27 18:50:59 dist-test-slave-8tg2 krb5kdc[18626](info): commencing operation
krb5kdc: starting...
W20260927 18:51:00.051716 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:00 dist-test-slave-8tg2 krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:51:00.172936 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:00.288735 18639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:00.289022 18639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:00.289065 18639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:00.293232 18639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:00.293304 18639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:00.293325 18639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:00.293349 18639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:00.293365 18639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:00.298906 18639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:00.300040 18639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:00.301050 18639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:00.309708 18645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:00.309798 18647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:00.309684 18644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:00.310091 18639 server_base.cc:1034] running on GCE node
I20260927 18:51:00.310654 18639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:00.311811 18639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:00.312980 18639 hybrid_clock.cc:648] HybridClock initialized: now 1790535060312967 us; error 26 us; skew 500 ppm
Sep 27 18:51:00 dist-test-slave-8tg2 krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:00.315714 18639 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:51:00.316646 18639 webserver.cc:460] Webserver started at http://127.0.0.1:35951/ using document root <none> and password file <none>
I20260927 18:51:00.317181 18639 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:00.317240 18639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:00.317476 18639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:00.319581 18639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "91977217d1474babbe2440ea0f2acce2"
format_stamp: "Formatted at 2026-09-27 18:51:00 on dist-test-slave-8tg2"
server_key: "828a9e8291977fc1ebe1fb713f8a0b43"
server_key_iv: "0a283c7516b3862755e63c10fb4c40cb"
server_key_version: "encryptionkey@0"
I20260927 18:51:00.320147 18639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "91977217d1474babbe2440ea0f2acce2"
format_stamp: "Formatted at 2026-09-27 18:51:00 on dist-test-slave-8tg2"
server_key: "828a9e8291977fc1ebe1fb713f8a0b43"
server_key_iv: "0a283c7516b3862755e63c10fb4c40cb"
server_key_version: "encryptionkey@0"
I20260927 18:51:00.323871 18639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 18:51:00.326225 18654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:00.327631 18639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:51:00.327755 18639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "91977217d1474babbe2440ea0f2acce2"
format_stamp: "Formatted at 2026-09-27 18:51:00 on dist-test-slave-8tg2"
server_key: "828a9e8291977fc1ebe1fb713f8a0b43"
server_key_iv: "0a283c7516b3862755e63c10fb4c40cb"
server_key_version: "encryptionkey@0"
I20260927 18:51:00.327857 18639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:00.332736 18639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:00.335814 18639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:00.335999 18639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:00.343236 18639 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42843
I20260927 18:51:00.343261 18706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42843 every 8 connection(s)
I20260927 18:51:00.344233 18639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:00.347033 18707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:00.349437 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 18639
I20260927 18:51:00.349563 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:00.349853 30333 external_mini_cluster.cc:1428] Setting key a8a0b4a8bbbd55ebc1cbd15b15a02169
I20260927 18:51:00.353327 18707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2: Bootstrap starting.
I20260927 18:51:00.355083 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:00.351319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53851 (local address 127.0.0.1:42843)
0927 18:51:00.351767 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:00.351782 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:00.351815 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:51:00.352451 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:00.352467 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:00.352762 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:00.353107 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:00.353908 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:00.353914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:00.353935 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:00.353945 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:00.353999 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:00.354018 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:00.354166 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:00.354236 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 18:51:00.354383 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":135,"threads_started":1}
I20260927 18:51:00.355782 18707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:00.356559 18707 log.cc:824] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:00.357918 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:00.358824 18707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2: No bootstrap required, opened a new log
I20260927 18:51:00.361831 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:00.360134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35257 (local address 127.0.0.1:42843)
0927 18:51:00.360253 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:00.360257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:00.360408 (+ 151us) server_negotiation.cc:407] Connection header received
0927 18:51:00.360507 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:00.360510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:00.360540 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:00.360613 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:00.361451 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:00.361454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:00.361457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:00.361468 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:00.361497 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:00.361511 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:00.361599 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:00.361668 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 18:51:00.361703 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:51:00.363008 18707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91977217d1474babbe2440ea0f2acce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } }
I20260927 18:51:00.363219 18707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:00.363293 18707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91977217d1474babbe2440ea0f2acce2, State: Initialized, Role: FOLLOWER
I20260927 18:51:00.363780 18707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91977217d1474babbe2440ea0f2acce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } }
I20260927 18:51:00.363900 18707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:00.363941 18707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:00.364032 18707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:00.365043 18707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91977217d1474babbe2440ea0f2acce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } }
I20260927 18:51:00.365377 18707 leader_election.cc:302] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91977217d1474babbe2440ea0f2acce2; no voters:
I20260927 18:51:00.365708 18707 leader_election.cc:288] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:00.365856 18717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:00.366093 18717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [term 1 LEADER]: Becoming Leader. State: Replica: 91977217d1474babbe2440ea0f2acce2, State: Running, Role: LEADER
I20260927 18:51:00.366463 18717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91977217d1474babbe2440ea0f2acce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } }
I20260927 18:51:00.366901 18707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:00.368142 18718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91977217d1474babbe2440ea0f2acce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91977217d1474babbe2440ea0f2acce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } } }
I20260927 18:51:00.368268 18718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:00.368634 18726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:00.369381 18719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91977217d1474babbe2440ea0f2acce2. Latest consensus state: current_term: 1 leader_uuid: "91977217d1474babbe2440ea0f2acce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91977217d1474babbe2440ea0f2acce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } } }
I20260927 18:51:00.369633 18719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:00.371776 18726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:00.377627 18726 catalog_manager.cc:1347] Generated new cluster ID: 8dbcdf2cffdb4e0da598b65371cb9343
I20260927 18:51:00.377709 18726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:00.391767 18726 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:00.392124 18726 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:00.405519 18726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91977217d1474babbe2440ea0f2acce2: Generated new TSK 0
I20260927 18:51:00.406328 18726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:00.416097 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 18639
2026-09-27T18:51:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:00 dist-test-slave-8tg2 krb5kdc[18738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:00 dist-test-slave-8tg2 krb5kdc[18738](info): set up 2 sockets
Sep 27 18:51:00 dist-test-slave-8tg2 krb5kdc[18738](info): commencing operation
krb5kdc: starting...
W20260927 18:51:01.451792 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:01 dist-test-slave-8tg2 krb5kdc[18738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:51:01.574833 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:01.684425 18751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:01.684748 18751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:01.684836 18751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:01.688944 18751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:01.689047 18751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:01.689086 18751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:01.689157 18751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:01.689196 18751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:01.694669 18751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:01.695868 18751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:01.696921 18751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:01.705714 18759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:01.705775 18751 server_base.cc:1034] running on GCE node
W20260927 18:51:01.705708 18757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:01.705782 18756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:01.706600 18751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:01.707775 18751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:01.708935 18751 hybrid_clock.cc:648] HybridClock initialized: now 1790535061708928 us; error 30 us; skew 500 ppm
Sep 27 18:51:01 dist-test-slave-8tg2 krb5kdc[18738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:01.711766 18751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:51:01.712698 18751 webserver.cc:460] Webserver started at http://127.0.0.1:33411/ using document root <none> and password file <none>
I20260927 18:51:01.713297 18751 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:01.713362 18751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:01.713609 18751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:01.715838 18751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "4accb5095ea049c18393dc1cfdfb30ae"
format_stamp: "Formatted at 2026-09-27 18:51:01 on dist-test-slave-8tg2"
server_key: "b60d7e2303e409b5f19f369323e6ba8b"
server_key_iv: "433beef7f47d643e3dcf8c3edbf18e05"
server_key_version: "encryptionkey@0"
I20260927 18:51:01.716424 18751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "4accb5095ea049c18393dc1cfdfb30ae"
format_stamp: "Formatted at 2026-09-27 18:51:01 on dist-test-slave-8tg2"
server_key: "b60d7e2303e409b5f19f369323e6ba8b"
server_key_iv: "433beef7f47d643e3dcf8c3edbf18e05"
server_key_version: "encryptionkey@0"
I20260927 18:51:01.720361 18751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:51:01.722913 18766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:01.724336 18751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:51:01.724478 18751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "4accb5095ea049c18393dc1cfdfb30ae"
format_stamp: "Formatted at 2026-09-27 18:51:01 on dist-test-slave-8tg2"
server_key: "b60d7e2303e409b5f19f369323e6ba8b"
server_key_iv: "433beef7f47d643e3dcf8c3edbf18e05"
server_key_version: "encryptionkey@0"
I20260927 18:51:01.724613 18751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:01.739490 18751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:01.742695 18751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:01.742910 18751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:01.750330 18751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44341
I20260927 18:51:01.750352 18818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44341 every 8 connection(s)
I20260927 18:51:01.751382 18751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:01.754204 18819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:01.759804 18819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae: Bootstrap starting.
I20260927 18:51:01.761355 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 18751
I20260927 18:51:01.761526 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:01.761818 30333 external_mini_cluster.cc:1428] Setting key 9c27540929ce239fdbb51cb909cc90a1
I20260927 18:51:01.762534 18819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:01.763522 18819 log.cc:824] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:01.765511 18819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae: No bootstrap required, opened a new log
I20260927 18:51:01.769176 18819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4accb5095ea049c18393dc1cfdfb30ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260927 18:51:01.769495 18819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:01.769582 18819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4accb5095ea049c18393dc1cfdfb30ae, State: Initialized, Role: FOLLOWER
I20260927 18:51:01.770054 18819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4accb5095ea049c18393dc1cfdfb30ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260927 18:51:01.770210 18819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:01.770299 18819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:01.770398 18819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [term 0 FOLLOWER]: Advancing to term 1
Sep 27 18:51:01 dist-test-slave-8tg2 krb5kdc[18738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:51:01.771728 18819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4accb5095ea049c18393dc1cfdfb30ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260927 18:51:01.772095 18819 leader_election.cc:302] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4accb5095ea049c18393dc1cfdfb30ae; no voters:
I20260927 18:51:01.772465 18819 leader_election.cc:288] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:01.772574 18824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:01.772744 18824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [term 1 LEADER]: Becoming Leader. State: Replica: 4accb5095ea049c18393dc1cfdfb30ae, State: Running, Role: LEADER
I20260927 18:51:01.773123 18824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4accb5095ea049c18393dc1cfdfb30ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } }
I20260927 18:51:01.773840 18819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:01.775125 18825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4accb5095ea049c18393dc1cfdfb30ae. Latest consensus state: current_term: 1 leader_uuid: "4accb5095ea049c18393dc1cfdfb30ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4accb5095ea049c18393dc1cfdfb30ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } } }
I20260927 18:51:01.775244 18825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:01.775230 18826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4accb5095ea049c18393dc1cfdfb30ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4accb5095ea049c18393dc1cfdfb30ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44341 } } }
I20260927 18:51:01.775722 18826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:01.775727 18834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:01.778628 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:01.763034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:38529 (local address 127.0.0.1:44341)
0927 18:51:01.763586 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:01.763603 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:01.763640 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:51:01.764396 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:01.764415 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:01.764775 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:01.765144 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:01.766001 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:01.766967 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:01.767938 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:01.768463 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:01.768647 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:01.772042 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:01.772048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:01.772065 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:01.772097 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:01.774520 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:01.775337 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:01.775344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:01.775350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:01.775552 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:01.776097 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:01.776101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:01.776103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:01.776627 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:01.776908 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:01.777227 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 18:51:01.777694 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":150,"threads_started":1}
I20260927 18:51:01.779289 18834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:01.781905 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:01.785981 18834 catalog_manager.cc:1347] Generated new cluster ID: 657bb92eb48e4d3cb97491f4cb7a616a
I20260927 18:51:01.786055 18834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:01.793362 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:01.784002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:42873 (local address 127.0.0.1:44341)
0927 18:51:01.784120 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:01.784125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:01.784301 (+ 176us) server_negotiation.cc:407] Connection header received
0927 18:51:01.784478 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:01.784481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:01.784531 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:01.784631 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:01.785377 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:01.786056 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:01.787249 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:01.787689 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:01.787796 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:01.789598 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:01.789602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:01.789605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:01.789632 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:01.791412 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:01.791984 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:01.791987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:01.791989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:01.792078 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:01.792424 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:01.792427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:01.792429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:01.792640 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:01.792781 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:01.793085 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 18:51:01.793190 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 18:51:01.800596 18834 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:01.801630 18834 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:01.806140 18834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4accb5095ea049c18393dc1cfdfb30ae: Generated new TSK 0
I20260927 18:51:01.806975 18834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:01.823230 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 18751
W20260927 18:51:01.828842 18736 connection.cc:535] client connection to 127.0.0.1:44341 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44341): Connection reset by peer (error 104)
2026-09-27T18:51:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:01 dist-test-slave-8tg2 krb5kdc[18850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:01 dist-test-slave-8tg2 krb5kdc[18850](info): set up 2 sockets
Sep 27 18:51:01 dist-test-slave-8tg2 krb5kdc[18850](info): commencing operation
krb5kdc: starting...
W20260927 18:51:02.883757 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:02 dist-test-slave-8tg2 krb5kdc[18850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:51:03.007277 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:03.121865 18863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:03.122216 18863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:03.122305 18863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:03.126617 18863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:03.126740 18863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:03.126780 18863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:03.126819 18863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:03.126849 18863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:03.132540 18863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:03.133781 18863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:03.134923 18863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:03.144321 18868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:03.144306 18871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:03.144357 18869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:03.144451 18863 server_base.cc:1034] running on GCE node
I20260927 18:51:03.145206 18863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:03.146350 18863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:03.147508 18863 hybrid_clock.cc:648] HybridClock initialized: now 1790535063147482 us; error 52 us; skew 500 ppm
Sep 27 18:51:03 dist-test-slave-8tg2 krb5kdc[18850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:03.157142 18863 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:51:03.158345 18863 webserver.cc:460] Webserver started at http://127.0.0.1:35133/ using document root <none> and password file <none>
I20260927 18:51:03.159041 18863 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:03.159144 18863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:03.159417 18863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:03.161778 18863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "fd897bffee5d477c9dbd1a93632ecf56"
format_stamp: "Formatted at 2026-09-27 18:51:03 on dist-test-slave-8tg2"
server_key: "6a19a9d9a7f0056145e7a7810ee0b6af"
server_key_iv: "fea245852d00d14ee1b60b8139f98ce3"
server_key_version: "encryptionkey@0"
I20260927 18:51:03.162418 18863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "fd897bffee5d477c9dbd1a93632ecf56"
format_stamp: "Formatted at 2026-09-27 18:51:03 on dist-test-slave-8tg2"
server_key: "6a19a9d9a7f0056145e7a7810ee0b6af"
server_key_iv: "fea245852d00d14ee1b60b8139f98ce3"
server_key_version: "encryptionkey@0"
I20260927 18:51:03.166529 18863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 18:51:03.169234 18878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:03.170583 18863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:51:03.170744 18863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "fd897bffee5d477c9dbd1a93632ecf56"
format_stamp: "Formatted at 2026-09-27 18:51:03 on dist-test-slave-8tg2"
server_key: "6a19a9d9a7f0056145e7a7810ee0b6af"
server_key_iv: "fea245852d00d14ee1b60b8139f98ce3"
server_key_version: "encryptionkey@0"
I20260927 18:51:03.170881 18863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:03.206087 18863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:03.209537 18863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:03.209774 18863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:03.217247 18863 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45557
I20260927 18:51:03.217262 18930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45557 every 8 connection(s)
I20260927 18:51:03.218250 18863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:03.221153 18931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:03.224373 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 18863
I20260927 18:51:03.224517 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:03.224823 30333 external_mini_cluster.cc:1428] Setting key 403383f38dda2f4b6fcd8dab24ca9c85
I20260927 18:51:03.227262 18931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56: Bootstrap starting.
I20260927 18:51:03.229853 18931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:03.230753 18931 log.cc:824] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:03.232707 18931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56: No bootstrap required, opened a new log
I20260927 18:51:03.234212 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:03.226238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:35751 (local address 127.0.0.1:45557)
0927 18:51:03.226645 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:03.226661 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:03.226698 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:51:03.227311 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:03.227328 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:03.227748 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:03.228034 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:03.228358 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:03.229322 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:03.230604 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:03.231168 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:03.231367 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:03.232425 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:03.232430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:03.232443 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:03.232894 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:03.232952 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:03.232975 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:03.233118 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:03.233308 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 18:51:03.233458 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":134,"threads_started":1}
I20260927 18:51:03.237003 18931 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd897bffee5d477c9dbd1a93632ecf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } }
I20260927 18:51:03.237260 18931 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:03.237335 18931 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd897bffee5d477c9dbd1a93632ecf56, State: Initialized, Role: FOLLOWER
I20260927 18:51:03.237489 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:03.237852 18931 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd897bffee5d477c9dbd1a93632ecf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } }
I20260927 18:51:03.237975 18931 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:03.238082 18931 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:03.238196 18931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:03.239217 18931 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd897bffee5d477c9dbd1a93632ecf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } }
I20260927 18:51:03.239636 18931 leader_election.cc:302] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd897bffee5d477c9dbd1a93632ecf56; no voters:
I20260927 18:51:03.239984 18931 leader_election.cc:288] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:03.240087 18941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:03.240339 18941 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [term 1 LEADER]: Becoming Leader. State: Replica: fd897bffee5d477c9dbd1a93632ecf56, State: Running, Role: LEADER
I20260927 18:51:03.240724 18941 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd897bffee5d477c9dbd1a93632ecf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } }
I20260927 18:51:03.241477 18931 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:03.241973 18943 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd897bffee5d477c9dbd1a93632ecf56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd897bffee5d477c9dbd1a93632ecf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } } }
I20260927 18:51:03.242714 18943 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:03.242584 18942 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd897bffee5d477c9dbd1a93632ecf56. Latest consensus state: current_term: 1 leader_uuid: "fd897bffee5d477c9dbd1a93632ecf56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd897bffee5d477c9dbd1a93632ecf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } } }
I20260927 18:51:03.243038 18942 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:03.244711 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:03.239708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:56939 (local address 127.0.0.1:45557)
0927 18:51:03.239831 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:03.239836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:03.240130 (+ 294us) server_negotiation.cc:407] Connection header received
0927 18:51:03.240185 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:03.240188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:03.240246 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:03.240345 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:03.240628 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:03.241400 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:03.242654 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:03.243063 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:03.243179 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:03.244279 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:03.244282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:03.244285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:03.244298 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:03.244324 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:03.244336 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:03.244425 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:03.244544 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 18:51:03.244578 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 18:51:03.246989 18956 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:51:03.247074 18956 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:51:03.247148 18950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:03.248019 18950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:03.253763 18950 catalog_manager.cc:1347] Generated new cluster ID: 1a992622d7514500ac82da4b8b339123
I20260927 18:51:03.253861 18950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:03.263640 18950 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:03.264679 18950 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:03.276041 18950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd897bffee5d477c9dbd1a93632ecf56: Generated new TSK 0
I20260927 18:51:03.276943 18950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:03.288856 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 18863
W20260927 18:51:03.294744 18848 connection.cc:535] client connection to 127.0.0.1:45557 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45557): Connection reset by peer (error 104)
2026-09-27T18:51:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:03 dist-test-slave-8tg2 krb5kdc[18962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:03 dist-test-slave-8tg2 krb5kdc[18962](info): set up 2 sockets
Sep 27 18:51:03 dist-test-slave-8tg2 krb5kdc[18962](info): commencing operation
krb5kdc: starting...
W20260927 18:51:04.331761 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:04 dist-test-slave-8tg2 krb5kdc[18962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:51:04.458148 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:04.567402 18975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:04.567695 18975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:04.567737 18975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:04.571671 18975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:04.571732 18975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:04.571753 18975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:04.571771 18975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:04.571786 18975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:04.577193 18975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:04.578236 18975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:04.579233 18975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:04.587705 18981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:04.587703 18980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:04.587703 18983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:04.588265 18975 server_base.cc:1034] running on GCE node
I20260927 18:51:04.588809 18975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:04.589924 18975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:04.591099 18975 hybrid_clock.cc:648] HybridClock initialized: now 1790535064591085 us; error 28 us; skew 500 ppm
Sep 27 18:51:04 dist-test-slave-8tg2 krb5kdc[18962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:04.593964 18975 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:51:04.594830 18975 webserver.cc:460] Webserver started at http://127.0.0.1:37043/ using document root <none> and password file <none>
I20260927 18:51:04.595391 18975 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:04.595532 18975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:04.595768 18975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:04.597792 18975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "92c597064b304e78b4cab93fb6d5782c"
format_stamp: "Formatted at 2026-09-27 18:51:04 on dist-test-slave-8tg2"
server_key: "e7cdd221b09b53c680087422cd7683a9"
server_key_iv: "58161b0b43da0bde2715f59ac119fc22"
server_key_version: "encryptionkey@0"
I20260927 18:51:04.598383 18975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "92c597064b304e78b4cab93fb6d5782c"
format_stamp: "Formatted at 2026-09-27 18:51:04 on dist-test-slave-8tg2"
server_key: "e7cdd221b09b53c680087422cd7683a9"
server_key_iv: "58161b0b43da0bde2715f59ac119fc22"
server_key_version: "encryptionkey@0"
I20260927 18:51:04.602218 18975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:51:04.604830 18990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:04.606182 18975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:51:04.606324 18975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "92c597064b304e78b4cab93fb6d5782c"
format_stamp: "Formatted at 2026-09-27 18:51:04 on dist-test-slave-8tg2"
server_key: "e7cdd221b09b53c680087422cd7683a9"
server_key_iv: "58161b0b43da0bde2715f59ac119fc22"
server_key_version: "encryptionkey@0"
I20260927 18:51:04.606443 18975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:04.611106 18975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:04.614090 18975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:04.614298 18975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:04.621457 18975 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44755
I20260927 18:51:04.621481 19042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44755 every 8 connection(s)
I20260927 18:51:04.622356 18975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:04.624171 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 18975
I20260927 18:51:04.624315 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:04.624640 30333 external_mini_cluster.cc:1428] Setting key cde7f80b9ab179ecaa225e08e75ca983
I20260927 18:51:04.625463 19043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:04.629809 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:04.626039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:52355 (local address 127.0.0.1:44755)
0927 18:51:04.626442 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:04.626456 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:04.626492 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:51:04.627137 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:04.627153 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:04.627480 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:04.627837 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:04.628650 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:04.628654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:04.628666 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:04.628676 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:04.628730 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:04.628748 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:04.628910 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:04.628980 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 18:51:04.629145 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":122,"threads_started":1}
I20260927 18:51:04.631759 19043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c: Bootstrap starting.
I20260927 18:51:04.632687 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:04.634162 19043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:04.634953 19043 log.cc:824] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:04.636545 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:04.634765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:49399 (local address 127.0.0.1:44755)
0927 18:51:04.634888 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:04.634893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:04.635077 (+ 184us) server_negotiation.cc:407] Connection header received
0927 18:51:04.635195 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:04.635197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:04.635223 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:04.635286 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:04.636116 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:04.636119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:04.636121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:04.636131 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:04.636153 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:04.636163 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:04.636229 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:04.636347 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 18:51:04.636377 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 18:51:04.637422 19043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c: No bootstrap required, opened a new log
I20260927 18:51:04.640909 19043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c597064b304e78b4cab93fb6d5782c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44755 } }
I20260927 18:51:04.641149 19043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:04.641232 19043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c597064b304e78b4cab93fb6d5782c, State: Initialized, Role: FOLLOWER
I20260927 18:51:04.641686 19043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c597064b304e78b4cab93fb6d5782c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44755 } }
I20260927 18:51:04.641840 19043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:04.641913 19043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:04.642019 19043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:04.642970 19043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c597064b304e78b4cab93fb6d5782c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44755 } }
I20260927 18:51:04.643311 19043 leader_election.cc:302] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92c597064b304e78b4cab93fb6d5782c; no voters:
I20260927 18:51:04.643643 19043 leader_election.cc:288] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:04.643991 19053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:04.644236 19053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [term 1 LEADER]: Becoming Leader. State: Replica: 92c597064b304e78b4cab93fb6d5782c, State: Running, Role: LEADER
I20260927 18:51:04.644554 19053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c597064b304e78b4cab93fb6d5782c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44755 } }
I20260927 18:51:04.645099 19043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:04.646255 19055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c597064b304e78b4cab93fb6d5782c. Latest consensus state: current_term: 1 leader_uuid: "92c597064b304e78b4cab93fb6d5782c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c597064b304e78b4cab93fb6d5782c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44755 } } }
I20260927 18:51:04.646389 19055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:04.646764 19062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:04.646718 19054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92c597064b304e78b4cab93fb6d5782c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c597064b304e78b4cab93fb6d5782c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44755 } } }
I20260927 18:51:04.647343 19054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:04.650144 19062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:04.655910 19062 catalog_manager.cc:1347] Generated new cluster ID: 15ee92c127dc4bb3bfa81a1fa462f242
I20260927 18:51:04.655982 19062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:04.696468 19062 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:04.696867 19062 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:04.703768 19062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92c597064b304e78b4cab93fb6d5782c: Generated new TSK 0
I20260927 18:51:04.704702 19062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:04.720593 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 18975
2026-09-27T18:51:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:04 dist-test-slave-8tg2 krb5kdc[19074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:04 dist-test-slave-8tg2 krb5kdc[19074](info): set up 2 sockets
Sep 27 18:51:04 dist-test-slave-8tg2 krb5kdc[19074](info): commencing operation
krb5kdc: starting...
W20260927 18:51:05.755823 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:05 dist-test-slave-8tg2 krb5kdc[19074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:51:05.876015 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:05.987799 19087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:05.988070 19087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:05.988111 19087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:05.992044 19087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:05.992122 19087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:05.992144 19087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:05.992161 19087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:05.992177 19087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:05.997313 19087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44139
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:05.998293 19087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:05.999302 19087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:06.007891 19093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:06.007918 19092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:06.008085 19087 server_base.cc:1034] running on GCE node
W20260927 18:51:06.007918 19095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:06.008810 19087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:06.009945 19087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:06.011114 19087 hybrid_clock.cc:648] HybridClock initialized: now 1790535066011064 us; error 64 us; skew 500 ppm
Sep 27 18:51:06 dist-test-slave-8tg2 krb5kdc[19074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:06.014031 19087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:51:06.014941 19087 webserver.cc:460] Webserver started at http://127.0.0.1:45713/ using document root <none> and password file <none>
I20260927 18:51:06.015558 19087 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:06.015625 19087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:06.015861 19087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:06.017933 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "53046ac5500d4b059c4d8ac1b14264d0"
format_stamp: "Formatted at 2026-09-27 18:51:06 on dist-test-slave-8tg2"
server_key: "61b886c0325257132248f39f210593b7"
server_key_iv: "18b3a43009da4541051d0bd88fbbe8d3"
server_key_version: "encryptionkey@0"
I20260927 18:51:06.018507 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "53046ac5500d4b059c4d8ac1b14264d0"
format_stamp: "Formatted at 2026-09-27 18:51:06 on dist-test-slave-8tg2"
server_key: "61b886c0325257132248f39f210593b7"
server_key_iv: "18b3a43009da4541051d0bd88fbbe8d3"
server_key_version: "encryptionkey@0"
I20260927 18:51:06.022270 19087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 18:51:06.024775 19102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:06.026094 19087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:51:06.026196 19087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "53046ac5500d4b059c4d8ac1b14264d0"
format_stamp: "Formatted at 2026-09-27 18:51:06 on dist-test-slave-8tg2"
server_key: "61b886c0325257132248f39f210593b7"
server_key_iv: "18b3a43009da4541051d0bd88fbbe8d3"
server_key_version: "encryptionkey@0"
I20260927 18:51:06.026284 19087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:06.057121 19087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:06.060201 19087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:06.060359 19087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:06.067226 19087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44139
I20260927 18:51:06.067255 19154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44139 every 8 connection(s)
I20260927 18:51:06.068204 19087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:06.070866 19155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:06.072522 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 19087
I20260927 18:51:06.072669 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:06.072940 30333 external_mini_cluster.cc:1428] Setting key 4b92acea18787d390862d9b50b2fb99d
I20260927 18:51:06.077591 19155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0: Bootstrap starting.
I20260927 18:51:06.080230 19155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:06.081104 19155 log.cc:824] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:51:06 dist-test-slave-8tg2 krb5kdc[19074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:51:06.083338 19155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0: No bootstrap required, opened a new log
I20260927 18:51:06.088171 19155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53046ac5500d4b059c4d8ac1b14264d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } }
I20260927 18:51:06.088425 19155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:06.088474 19155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53046ac5500d4b059c4d8ac1b14264d0, State: Initialized, Role: FOLLOWER
I20260927 18:51:06.088689 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:06.074339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:38601 (local address 127.0.0.1:44139)
0927 18:51:06.074749 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:06.074767 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:06.074808 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:51:06.075711 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:06.075735 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:06.076173 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:06.076547 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:06.077221 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:06.078640 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:06.079623 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:06.080318 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:06.080489 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:06.083370 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:06.083378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:06.083395 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:06.083443 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:06.085517 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:06.086060 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:06.086066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:06.086073 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:06.086209 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:06.086519 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:06.086529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:06.086531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:06.087021 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:06.087247 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:06.087488 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 18:51:06.087727 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":102,"threads_started":1}
I20260927 18:51:06.088937 19155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53046ac5500d4b059c4d8ac1b14264d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } }
I20260927 18:51:06.089114 19155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:06.089181 19155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:06.089273 19155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:06.090160 19155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53046ac5500d4b059c4d8ac1b14264d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } }
I20260927 18:51:06.090498 19155 leader_election.cc:302] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53046ac5500d4b059c4d8ac1b14264d0; no voters:
I20260927 18:51:06.090828 19155 leader_election.cc:288] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:06.090929 19160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:06.091157 19160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [term 1 LEADER]: Becoming Leader. State: Replica: 53046ac5500d4b059c4d8ac1b14264d0, State: Running, Role: LEADER
I20260927 18:51:06.091555 19160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53046ac5500d4b059c4d8ac1b14264d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } }
I20260927 18:51:06.091758 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:06.092177 19155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:06.093209 19162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53046ac5500d4b059c4d8ac1b14264d0. Latest consensus state: current_term: 1 leader_uuid: "53046ac5500d4b059c4d8ac1b14264d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53046ac5500d4b059c4d8ac1b14264d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } } }
I20260927 18:51:06.093329 19162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:06.093551 19161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53046ac5500d4b059c4d8ac1b14264d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53046ac5500d4b059c4d8ac1b14264d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } } }
I20260927 18:51:06.093657 19161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:06.093806 19173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:06.097146 19173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:06.103101 19173 catalog_manager.cc:1347] Generated new cluster ID: 071204acc1b94c17bb9f045e48ab485c
I20260927 18:51:06.103232 19173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:06.103929 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:06.094267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:50503 (local address 127.0.0.1:44139)
0927 18:51:06.094380 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:06.094386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:06.094595 (+ 209us) server_negotiation.cc:407] Connection header received
0927 18:51:06.094731 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:06.094734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:06.094779 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:06.094863 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:06.095782 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:06.096546 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:06.097637 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:06.097973 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:06.098082 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:06.099787 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:06.099790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:06.099793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:06.099825 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:06.101298 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:06.101994 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:06.101998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:06.102000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:06.102088 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:06.102582 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:06.102585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:06.102587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:06.102824 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:06.102973 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:06.103674 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 18:51:06.103773 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 18:51:06.119488 19173 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:06.120842 19173 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:06.141399 19173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53046ac5500d4b059c4d8ac1b14264d0: Generated new TSK 0
I20260927 18:51:06.142323 19173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:06.155612 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 19087
W20260927 18:51:06.160490 19072 connection.cc:535] client connection to 127.0.0.1:44139 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44139): Connection reset by peer (error 104)
2026-09-27T18:51:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:06 dist-test-slave-8tg2 krb5kdc[19186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:06 dist-test-slave-8tg2 krb5kdc[19186](info): set up 2 sockets
Sep 27 18:51:06 dist-test-slave-8tg2 krb5kdc[19186](info): commencing operation
krb5kdc: starting...
W20260927 18:51:07.191851 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:07 dist-test-slave-8tg2 krb5kdc[19186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:51:07.318042 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:07.432485 19199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:07.432781 19199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:07.432825 19199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:07.436908 19199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:07.436987 19199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:07.437008 19199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:07.437026 19199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:07.437041 19199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:07.443111 19199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:07.444247 19199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:07.445271 19199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:07.454047 19204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:07.454056 19207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:07.454057 19205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:07.454397 19199 server_base.cc:1034] running on GCE node
I20260927 18:51:07.454993 19199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:07.456156 19199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:07.457319 19199 hybrid_clock.cc:648] HybridClock initialized: now 1790535067457291 us; error 61 us; skew 500 ppm
Sep 27 18:51:07 dist-test-slave-8tg2 krb5kdc[19186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:07.460144 19199 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:51:07.461087 19199 webserver.cc:460] Webserver started at http://127.0.0.1:42265/ using document root <none> and password file <none>
I20260927 18:51:07.461633 19199 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:07.461694 19199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:07.461861 19199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:07.463981 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "eceb627fb48748d395fe58504e98a0fe"
format_stamp: "Formatted at 2026-09-27 18:51:07 on dist-test-slave-8tg2"
server_key: "1349ff10efd9dd1a37db1aaa143c53c1"
server_key_iv: "10e14493c86bf6550b1aca5781b34bc7"
server_key_version: "encryptionkey@0"
I20260927 18:51:07.464529 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "eceb627fb48748d395fe58504e98a0fe"
format_stamp: "Formatted at 2026-09-27 18:51:07 on dist-test-slave-8tg2"
server_key: "1349ff10efd9dd1a37db1aaa143c53c1"
server_key_iv: "10e14493c86bf6550b1aca5781b34bc7"
server_key_version: "encryptionkey@0"
I20260927 18:51:07.468361 19199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:51:07.470844 19214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:07.472173 19199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:51:07.472327 19199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "eceb627fb48748d395fe58504e98a0fe"
format_stamp: "Formatted at 2026-09-27 18:51:07 on dist-test-slave-8tg2"
server_key: "1349ff10efd9dd1a37db1aaa143c53c1"
server_key_iv: "10e14493c86bf6550b1aca5781b34bc7"
server_key_version: "encryptionkey@0"
I20260927 18:51:07.472452 19199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:07.483932 19199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:07.486949 19199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:07.487177 19199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:07.494553 19199 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35443
I20260927 18:51:07.494573 19266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35443 every 8 connection(s)
I20260927 18:51:07.495672 19199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:07.498625 19267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:07.504348 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 19199
I20260927 18:51:07.504499 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:07.504671 19267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe: Bootstrap starting.
I20260927 18:51:07.504813 30333 external_mini_cluster.cc:1428] Setting key 3963d53ac5f3f7301df130803e1679eb
I20260927 18:51:07.507367 19267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:07.508167 19267 log.cc:824] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:07.510082 19267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe: No bootstrap required, opened a new log
I20260927 18:51:07.514106 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:07.506249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:41117 (local address 127.0.0.1:35443)
0927 18:51:07.506726 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:07.506742 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:07.506778 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:51:07.507516 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:07.507543 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:07.507916 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:07.508261 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:07.508699 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:07.509670 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:07.510661 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:07.511173 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:07.511365 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:07.512332 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:07.512337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:07.512352 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:07.512756 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:07.512810 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:07.512834 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:07.512982 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:07.513183 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 18:51:07.513328 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":142,"threads_started":1}
I20260927 18:51:07.514036 19267 raft_consensus.cc:348] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eceb627fb48748d395fe58504e98a0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } }
I20260927 18:51:07.514343 19267 raft_consensus.cc:374] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:07.514439 19267 raft_consensus.cc:729] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eceb627fb48748d395fe58504e98a0fe, State: Initialized, Role: FOLLOWER
I20260927 18:51:07.514955 19267 consensus_queue.cc:260] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eceb627fb48748d395fe58504e98a0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } }
I20260927 18:51:07.515115 19267 raft_consensus.cc:388] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:07.515182 19267 raft_consensus.cc:482] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:07.515287 19267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:07.516520 19267 raft_consensus.cc:504] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eceb627fb48748d395fe58504e98a0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } }
I20260927 18:51:07.516880 19267 leader_election.cc:302] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eceb627fb48748d395fe58504e98a0fe; no voters:
I20260927 18:51:07.517261 19267 leader_election.cc:288] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:07.517344 19272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:07.517554 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:07.517612 19272 raft_consensus.cc:686] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [term 1 LEADER]: Becoming Leader. State: Replica: eceb627fb48748d395fe58504e98a0fe, State: Running, Role: LEADER
I20260927 18:51:07.517920 19272 consensus_queue.cc:237] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eceb627fb48748d395fe58504e98a0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } }
I20260927 18:51:07.518743 19267 sys_catalog.cc:564] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:07.519012 19272 sys_catalog.cc:455] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader eceb627fb48748d395fe58504e98a0fe. Latest consensus state: current_term: 1 leader_uuid: "eceb627fb48748d395fe58504e98a0fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eceb627fb48748d395fe58504e98a0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } } }
I20260927 18:51:07.519125 19272 sys_catalog.cc:458] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:07.519598 19273 sys_catalog.cc:455] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eceb627fb48748d395fe58504e98a0fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eceb627fb48748d395fe58504e98a0fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35443 } } }
I20260927 18:51:07.519759 19273 sys_catalog.cc:458] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:07.520625 19280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:07.524050 19280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:07.525238 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:07.520117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:49321 (local address 127.0.0.1:35443)
0927 18:51:07.520261 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:07.520266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:07.520536 (+ 270us) server_negotiation.cc:407] Connection header received
0927 18:51:07.520739 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:07.520742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:07.520790 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:07.520842 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:07.521266 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:07.522032 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:07.523177 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:07.523570 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:07.523676 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:07.524705 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:07.524709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:07.524712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:07.524726 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:07.524756 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:07.524771 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:07.524838 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:07.525054 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 18:51:07.525090 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:51:07.530460 19280 catalog_manager.cc:1347] Generated new cluster ID: 2a7989f01dff4340b2f0f9ac23270d9f
I20260927 18:51:07.530551 19280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:07.545693 19280 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:07.546695 19280 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:07.554201 19280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eceb627fb48748d395fe58504e98a0fe: Generated new TSK 0
I20260927 18:51:07.554904 19280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:07.560847 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 19199
W20260927 18:51:07.566334 19184 connection.cc:535] client connection to 127.0.0.1:35443 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35443): Connection reset by peer (error 104)
2026-09-27T18:51:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:07 dist-test-slave-8tg2 krb5kdc[19298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:07 dist-test-slave-8tg2 krb5kdc[19298](info): set up 2 sockets
Sep 27 18:51:07 dist-test-slave-8tg2 krb5kdc[19298](info): commencing operation
krb5kdc: starting...
W20260927 18:51:08.603688 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:08 dist-test-slave-8tg2 krb5kdc[19298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:51:08.731185 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:08.847838 19311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:08.848125 19311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:08.848170 19311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:08.852195 19311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:08.852257 19311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:08.852276 19311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:08.852293 19311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:08.852362 19311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:08.857990 19311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:08.859195 19311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:08.860443 19311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:08.869318 19317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:08.869324 19316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:08.869510 19311 server_base.cc:1034] running on GCE node
W20260927 18:51:08.869326 19319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:08.870247 19311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:08.871368 19311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:08.872589 19311 hybrid_clock.cc:648] HybridClock initialized: now 1790535068872574 us; error 28 us; skew 500 ppm
Sep 27 18:51:08 dist-test-slave-8tg2 krb5kdc[19298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:08.881470 19311 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:51:08.882541 19311 webserver.cc:460] Webserver started at http://127.0.0.1:35971/ using document root <none> and password file <none>
I20260927 18:51:08.883153 19311 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:08.883260 19311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:08.883577 19311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:08.885744 19311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "81e6d02dd28b449d8707009fdc071f94"
format_stamp: "Formatted at 2026-09-27 18:51:08 on dist-test-slave-8tg2"
server_key: "083b25576856582d8ff85dcaf8710ad5"
server_key_iv: "6f641282dac111c947d7a3f01de46314"
server_key_version: "encryptionkey@0"
I20260927 18:51:08.886395 19311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "81e6d02dd28b449d8707009fdc071f94"
format_stamp: "Formatted at 2026-09-27 18:51:08 on dist-test-slave-8tg2"
server_key: "083b25576856582d8ff85dcaf8710ad5"
server_key_iv: "6f641282dac111c947d7a3f01de46314"
server_key_version: "encryptionkey@0"
I20260927 18:51:08.890491 19311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 18:51:08.893167 19326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:08.894500 19311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:51:08.894672 19311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "81e6d02dd28b449d8707009fdc071f94"
format_stamp: "Formatted at 2026-09-27 18:51:08 on dist-test-slave-8tg2"
server_key: "083b25576856582d8ff85dcaf8710ad5"
server_key_iv: "6f641282dac111c947d7a3f01de46314"
server_key_version: "encryptionkey@0"
I20260927 18:51:08.894801 19311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:08.899959 19311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:08.903131 19311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:08.903353 19311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:08.910485 19311 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36257
I20260927 18:51:08.910504 19378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36257 every 8 connection(s)
I20260927 18:51:08.911619 19311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:08.914518 19379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:08.918206 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 19311
I20260927 18:51:08.918393 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:08.918670 30333 external_mini_cluster.cc:1428] Setting key 22110f7d427c7207a5d277e0d25b20ff
I20260927 18:51:08.920671 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94: Bootstrap starting.
I20260927 18:51:08.923484 19379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:08.924299 19379 log.cc:824] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:08.924343 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:08.920137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:48723 (local address 127.0.0.1:36257)
0927 18:51:08.920640 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:08.920655 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:08.920692 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:51:08.921375 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:08.921393 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:08.921703 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:08.922007 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:08.922896 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:08.922901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:08.922916 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:08.922928 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:08.922989 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:08.923011 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:08.923178 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:08.923325 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 18:51:08.923577 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":146,"threads_started":1}
I20260927 18:51:08.926479 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94: No bootstrap required, opened a new log
I20260927 18:51:08.927302 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:08.930923 19379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6d02dd28b449d8707009fdc071f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260927 18:51:08.931210 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:08.929512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:34031 (local address 127.0.0.1:36257)
0927 18:51:08.929632 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:08.929636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:08.929811 (+ 175us) server_negotiation.cc:407] Connection header received
0927 18:51:08.929936 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:08.929938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:08.929968 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:08.930034 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:08.930841 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:08.930844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:08.930847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:08.930858 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:08.930888 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:08.930903 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:08.930998 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:08.931059 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 18:51:08.931100 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:51:08.931250 19379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:08.931300 19379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e6d02dd28b449d8707009fdc071f94, State: Initialized, Role: FOLLOWER
I20260927 18:51:08.931762 19379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6d02dd28b449d8707009fdc071f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260927 18:51:08.931917 19379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:08.931983 19379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:08.932094 19379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:08.933048 19379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6d02dd28b449d8707009fdc071f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260927 18:51:08.933403 19379 leader_election.cc:302] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e6d02dd28b449d8707009fdc071f94; no voters:
I20260927 18:51:08.933743 19379 leader_election.cc:288] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:08.933823 19389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:08.934103 19389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [term 1 LEADER]: Becoming Leader. State: Replica: 81e6d02dd28b449d8707009fdc071f94, State: Running, Role: LEADER
I20260927 18:51:08.934464 19389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6d02dd28b449d8707009fdc071f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } }
I20260927 18:51:08.935103 19379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:08.936252 19390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81e6d02dd28b449d8707009fdc071f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6d02dd28b449d8707009fdc071f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } } }
I20260927 18:51:08.936380 19390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:08.936667 19391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e6d02dd28b449d8707009fdc071f94. Latest consensus state: current_term: 1 leader_uuid: "81e6d02dd28b449d8707009fdc071f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e6d02dd28b449d8707009fdc071f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36257 } } }
I20260927 18:51:08.936792 19391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:08.936836 19398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:08.940089 19398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:08.945657 19398 catalog_manager.cc:1347] Generated new cluster ID: 96c3e2b3684d4728bda9ecf61f17d8cc
I20260927 18:51:08.945811 19398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:08.969599 19398 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:08.969987 19398 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:08.981010 19398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81e6d02dd28b449d8707009fdc071f94: Generated new TSK 0
I20260927 18:51:08.981884 19398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:08.984962 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 19311
2026-09-27T18:51:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:09 dist-test-slave-8tg2 krb5kdc[19410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:09 dist-test-slave-8tg2 krb5kdc[19410](info): set up 2 sockets
Sep 27 18:51:09 dist-test-slave-8tg2 krb5kdc[19410](info): commencing operation
krb5kdc: starting...
W20260927 18:51:10.023837 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:10 dist-test-slave-8tg2 krb5kdc[19410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:51:10.151261 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:40301
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:40301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:10.276387 19423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:10.276706 19423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:10.276793 19423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:10.281196 19423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:10.281277 19423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:10.281297 19423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:10.281352 19423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:10.281387 19423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:10.287300 19423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:40301
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:40301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:10.288553 19423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:10.289675 19423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:10.298794 19429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:10.298826 19431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:10.299013 19428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:10.299366 19423 server_base.cc:1034] running on GCE node
I20260927 18:51:10.299930 19423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:10.301105 19423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:10.302282 19423 hybrid_clock.cc:648] HybridClock initialized: now 1790535070302264 us; error 34 us; skew 500 ppm
Sep 27 18:51:10 dist-test-slave-8tg2 krb5kdc[19410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535070, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:10.305467 19423 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:51:10.306384 19423 webserver.cc:460] Webserver started at http://127.29.159.126:36707/ using document root <none> and password file <none>
I20260927 18:51:10.306974 19423 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:10.307041 19423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:10.307292 19423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:10.309423 19423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "db60436e5ddb4bf5ba8f62b87398a347"
format_stamp: "Formatted at 2026-09-27 18:51:10 on dist-test-slave-8tg2"
server_key: "4602b885cbb20562564a0a2f63f8cbbe"
server_key_iv: "78f6c26011291e2bfc1f960a80e79f19"
server_key_version: "encryptionkey@0"
I20260927 18:51:10.310046 19423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "db60436e5ddb4bf5ba8f62b87398a347"
format_stamp: "Formatted at 2026-09-27 18:51:10 on dist-test-slave-8tg2"
server_key: "4602b885cbb20562564a0a2f63f8cbbe"
server_key_iv: "78f6c26011291e2bfc1f960a80e79f19"
server_key_version: "encryptionkey@0"
I20260927 18:51:10.314128 19423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:51:10.316814 19438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:10.318202 19423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:51:10.318378 19423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "db60436e5ddb4bf5ba8f62b87398a347"
format_stamp: "Formatted at 2026-09-27 18:51:10 on dist-test-slave-8tg2"
server_key: "4602b885cbb20562564a0a2f63f8cbbe"
server_key_iv: "78f6c26011291e2bfc1f960a80e79f19"
server_key_version: "encryptionkey@0"
I20260927 18:51:10.318522 19423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:10.334308 19423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:10.337632 19423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:10.337862 19423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:10.345263 19423 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:40301
I20260927 18:51:10.345281 19490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:40301 every 8 connection(s)
I20260927 18:51:10.346267 19423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:10.348313 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 19423
I20260927 18:51:10.348459 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:10.348767 30333 external_mini_cluster.cc:1428] Setting key 6c2892afe1982f487c60200549d2e194
I20260927 18:51:10.349539 19491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:10.356690 19491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347: Bootstrap starting.
Sep 27 18:51:10 dist-test-slave-8tg2 krb5kdc[19410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:51:10.359477 19491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:10.360435 19491 log.cc:824] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:10.362350 19491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347: No bootstrap required, opened a new log
I20260927 18:51:10.364356 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:10.350216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38681 (local address 127.29.159.126:40301)
0927 18:51:10.350618 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:10.350642 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:10.350684 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:51:10.351484 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:10.351509 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:10.351878 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:10.352171 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:10.353114 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:10.354111 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:10.355313 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:10.356186 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:10.356320 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:10.359286 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:10.359291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:10.359305 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:10.359335 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:10.361184 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:10.361728 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:10.361733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:10.361738 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:10.361846 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:10.362187 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:10.362192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:10.362199 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:10.362649 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:10.362868 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:10.363195 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 18:51:10.363415 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":141,"threads_started":1}
I20260927 18:51:10.366441 19491 raft_consensus.cc:348] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db60436e5ddb4bf5ba8f62b87398a347" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40301 } }
I20260927 18:51:10.366711 19491 raft_consensus.cc:374] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:10.366784 19491 raft_consensus.cc:729] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db60436e5ddb4bf5ba8f62b87398a347, State: Initialized, Role: FOLLOWER
I20260927 18:51:10.367210 19491 consensus_queue.cc:260] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db60436e5ddb4bf5ba8f62b87398a347" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40301 } }
I20260927 18:51:10.367378 19491 raft_consensus.cc:388] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:10.367446 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:10.367466 19491 raft_consensus.cc:482] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:10.367586 19491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:10.368528 19491 raft_consensus.cc:504] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db60436e5ddb4bf5ba8f62b87398a347" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40301 } }
I20260927 18:51:10.368887 19491 leader_election.cc:302] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db60436e5ddb4bf5ba8f62b87398a347; no voters:
I20260927 18:51:10.369189 19491 leader_election.cc:288] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:10.369283 19500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:10.369530 19500 raft_consensus.cc:686] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [term 1 LEADER]: Becoming Leader. State: Replica: db60436e5ddb4bf5ba8f62b87398a347, State: Running, Role: LEADER
I20260927 18:51:10.369933 19500 consensus_queue.cc:237] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db60436e5ddb4bf5ba8f62b87398a347" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40301 } }
I20260927 18:51:10.370467 19491 sys_catalog.cc:564] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:10.371270 19500 sys_catalog.cc:455] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db60436e5ddb4bf5ba8f62b87398a347. Latest consensus state: current_term: 1 leader_uuid: "db60436e5ddb4bf5ba8f62b87398a347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db60436e5ddb4bf5ba8f62b87398a347" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40301 } } }
I20260927 18:51:10.371383 19500 sys_catalog.cc:458] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:10.371747 19501 sys_catalog.cc:455] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db60436e5ddb4bf5ba8f62b87398a347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db60436e5ddb4bf5ba8f62b87398a347" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40301 } } }
I20260927 18:51:10.371865 19501 sys_catalog.cc:458] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347 [sys.catalog]: This master's current role is: LEADER
W20260927 18:51:10.375712 19516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:51:10.375806 19516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:51:10.376237 19509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:10.377188 19509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:10.381083 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:10.369724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59233 (local address 127.29.159.126:40301)
0927 18:51:10.369844 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:10.369848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:10.371835 (+ 1987us) server_negotiation.cc:407] Connection header received
0927 18:51:10.372125 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:10.372131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:10.372189 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:10.372287 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:10.373206 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:10.373745 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:10.374876 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:10.375232 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:10.375335 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:10.377043 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:10.377047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:10.377050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:10.377076 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:10.378964 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:10.379629 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:10.379633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:10.379635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:10.379727 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:10.380109 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:10.380113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:10.380114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:10.380337 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:10.380508 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:10.380814 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 18:51:10.380917 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:51:10.383050 19509 catalog_manager.cc:1347] Generated new cluster ID: ba442b8263734d20967064a2b78f9b56
I20260927 18:51:10.383112 19509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:10.410616 19509 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:10.411695 19509 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:10.421159 19509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db60436e5ddb4bf5ba8f62b87398a347: Generated new TSK 0
I20260927 18:51:10.421885 19509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:10.429258 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 19423
W20260927 18:51:10.434767 19408 connection.cc:535] client connection to 127.29.159.126:40301 recv error: Network error: failed to read from TLS socket (remote: 127.29.159.126:40301): Connection reset by peer (error 104)
2026-09-27T18:51:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:10 dist-test-slave-8tg2 krb5kdc[19522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:10 dist-test-slave-8tg2 krb5kdc[19522](info): set up 2 sockets
Sep 27 18:51:10 dist-test-slave-8tg2 krb5kdc[19522](info): commencing operation
krb5kdc: starting...
W20260927 18:51:11.475718 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:11 dist-test-slave-8tg2 krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:51:11.602591 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:34887
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:34887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:11.713188 19535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:11.713539 19535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:11.713632 19535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:11.717882 19535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:11.717983 19535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:11.718020 19535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:11.718082 19535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:11.718122 19535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:11.723593 19535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:34887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:34887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:11.724684 19535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:11.725723 19535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:11.734195 19543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:11.734247 19541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:11.734320 19535 server_base.cc:1034] running on GCE node
W20260927 18:51:11.734642 19540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:11.735206 19535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:11.736352 19535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:11.737557 19535 hybrid_clock.cc:648] HybridClock initialized: now 1790535071737544 us; error 25 us; skew 500 ppm
Sep 27 18:51:11 dist-test-slave-8tg2 krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535071, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:11.740424 19535 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:51:11.741362 19535 webserver.cc:460] Webserver started at http://127.29.159.126:33283/ using document root <none> and password file <none>
I20260927 18:51:11.741943 19535 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:11.742038 19535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:11.742280 19535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:11.744411 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "700c0b6ea33c49a8b1a6fc6feeba966f"
format_stamp: "Formatted at 2026-09-27 18:51:11 on dist-test-slave-8tg2"
server_key: "d77ea9aa38f591bd277bdafbae69a2c3"
server_key_iv: "ba56d3d9a4f392ce61793c7320d1e2c7"
server_key_version: "encryptionkey@0"
I20260927 18:51:11.744992 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "700c0b6ea33c49a8b1a6fc6feeba966f"
format_stamp: "Formatted at 2026-09-27 18:51:11 on dist-test-slave-8tg2"
server_key: "d77ea9aa38f591bd277bdafbae69a2c3"
server_key_iv: "ba56d3d9a4f392ce61793c7320d1e2c7"
server_key_version: "encryptionkey@0"
I20260927 18:51:11.748785 19535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:51:11.751281 19550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:11.752635 19535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:51:11.752794 19535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "700c0b6ea33c49a8b1a6fc6feeba966f"
format_stamp: "Formatted at 2026-09-27 18:51:11 on dist-test-slave-8tg2"
server_key: "d77ea9aa38f591bd277bdafbae69a2c3"
server_key_iv: "ba56d3d9a4f392ce61793c7320d1e2c7"
server_key_version: "encryptionkey@0"
I20260927 18:51:11.752916 19535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:11.772840 19535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:11.776611 19535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:11.776835 19535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:11.784150 19535 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:34887
I20260927 18:51:11.784176 19602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:34887 every 8 connection(s)
I20260927 18:51:11.785111 19535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:11.787779 19603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:11.788620 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 19535
I20260927 18:51:11.788766 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:11.789073 30333 external_mini_cluster.cc:1428] Setting key fd54838012dfbb970d51f0d1844388e9
I20260927 18:51:11.794278 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f: Bootstrap starting.
I20260927 18:51:11.797067 19603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:11.797899 19603 log.cc:824] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:11.799029 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:11.790608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35669 (local address 127.29.159.126:34887)
0927 18:51:11.791062 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:11.791078 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:11.791114 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:51:11.791876 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:11.791897 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:11.792264 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:11.792557 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:11.792959 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:11.794020 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:11.795287 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:11.795880 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:11.796011 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:11.796966 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:11.796971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:11.796985 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:11.797471 (+ 486us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:11.797532 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:11.797557 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:11.797666 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:11.797920 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 18:51:11.798071 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":135,"threads_started":1}
I20260927 18:51:11.800000 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f: No bootstrap required, opened a new log
I20260927 18:51:11.802001 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:11.804244 19603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c0b6ea33c49a8b1a6fc6feeba966f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34887 } }
I20260927 18:51:11.804638 19603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:11.804715 19603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700c0b6ea33c49a8b1a6fc6feeba966f, State: Initialized, Role: FOLLOWER
I20260927 18:51:11.805265 19603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c0b6ea33c49a8b1a6fc6feeba966f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34887 } }
I20260927 18:51:11.805431 19603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:11.805526 19603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:11.805625 19603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:11.806792 19603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c0b6ea33c49a8b1a6fc6feeba966f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34887 } }
I20260927 18:51:11.807171 19603 leader_election.cc:302] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 700c0b6ea33c49a8b1a6fc6feeba966f; no voters:
I20260927 18:51:11.807546 19603 leader_election.cc:288] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:11.807614 19613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:11.807850 19613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [term 1 LEADER]: Becoming Leader. State: Replica: 700c0b6ea33c49a8b1a6fc6feeba966f, State: Running, Role: LEADER
I20260927 18:51:11.808214 19613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c0b6ea33c49a8b1a6fc6feeba966f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34887 } }
I20260927 18:51:11.808909 19603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:11.809116 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:11.804417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43419 (local address 127.29.159.126:34887)
0927 18:51:11.804551 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:11.804556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:11.804768 (+ 212us) server_negotiation.cc:407] Connection header received
0927 18:51:11.804901 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:11.804904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:11.804944 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:11.805045 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:11.805333 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:11.806021 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:11.807154 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:11.807492 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:11.807623 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:11.808604 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:11.808607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:11.808610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:11.808623 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:11.808656 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:11.808667 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:11.808763 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:11.808923 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 18:51:11.808961 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:51:11.809423 19613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "700c0b6ea33c49a8b1a6fc6feeba966f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c0b6ea33c49a8b1a6fc6feeba966f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34887 } } }
I20260927 18:51:11.809602 19613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:11.810042 19619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:11.810002 19614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 700c0b6ea33c49a8b1a6fc6feeba966f. Latest consensus state: current_term: 1 leader_uuid: "700c0b6ea33c49a8b1a6fc6feeba966f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c0b6ea33c49a8b1a6fc6feeba966f" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 34887 } } }
I20260927 18:51:11.810230 19614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:11.813444 19619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:11.818578 19619 catalog_manager.cc:1347] Generated new cluster ID: 401db2f7d97444fca8c0c12f3c01d94d
I20260927 18:51:11.818643 19619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:11.836844 19619 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:11.837821 19619 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:11.841818 19619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 700c0b6ea33c49a8b1a6fc6feeba966f: Generated new TSK 0
I20260927 18:51:11.842514 19619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:11.848517 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 19535
W20260927 18:51:11.853865 19520 connection.cc:535] client connection to 127.29.159.126:34887 recv error: Network error: failed to read from TLS socket (remote: 127.29.159.126:34887): Connection reset by peer (error 104)
2026-09-27T18:51:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:11 dist-test-slave-8tg2 krb5kdc[19634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:11 dist-test-slave-8tg2 krb5kdc[19634](info): set up 2 sockets
Sep 27 18:51:11 dist-test-slave-8tg2 krb5kdc[19634](info): commencing operation
krb5kdc: starting...
W20260927 18:51:12.911793 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:12 dist-test-slave-8tg2 krb5kdc[19634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:51:13.035068 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:32967
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:32967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:13.146694 19647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:13.147009 19647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:13.147086 19647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:13.151031 19647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:13.151129 19647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:13.151168 19647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:13.151206 19647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:13.151242 19647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:13.156488 19647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:32967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:32967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:13.157575 19647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:13.158655 19647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:13.167017 19655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:13.166999 19652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:13.167066 19653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:13.167527 19647 server_base.cc:1034] running on GCE node
I20260927 18:51:13.168085 19647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:13.169214 19647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:13.170429 19647 hybrid_clock.cc:648] HybridClock initialized: now 1790535073170420 us; error 30 us; skew 500 ppm
Sep 27 18:51:13 dist-test-slave-8tg2 krb5kdc[19634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535073, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:13.173368 19647 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:51:13.174185 19647 webserver.cc:460] Webserver started at http://127.29.159.126:45591/ using document root <none> and password file <none>
I20260927 18:51:13.174690 19647 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:13.174751 19647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:13.174924 19647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:13.176944 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "6cfd361b3bf14ac2b2ec19dcbb2f3362"
format_stamp: "Formatted at 2026-09-27 18:51:13 on dist-test-slave-8tg2"
server_key: "9a7710e240c0425bb528f18187e1a68d"
server_key_iv: "db972e2a5e0a27b7524a1ef20c43e120"
server_key_version: "encryptionkey@0"
I20260927 18:51:13.177459 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "6cfd361b3bf14ac2b2ec19dcbb2f3362"
format_stamp: "Formatted at 2026-09-27 18:51:13 on dist-test-slave-8tg2"
server_key: "9a7710e240c0425bb528f18187e1a68d"
server_key_iv: "db972e2a5e0a27b7524a1ef20c43e120"
server_key_version: "encryptionkey@0"
I20260927 18:51:13.181141 19647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:51:13.183559 19662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:13.184834 19647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:51:13.184979 19647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "6cfd361b3bf14ac2b2ec19dcbb2f3362"
format_stamp: "Formatted at 2026-09-27 18:51:13 on dist-test-slave-8tg2"
server_key: "9a7710e240c0425bb528f18187e1a68d"
server_key_iv: "db972e2a5e0a27b7524a1ef20c43e120"
server_key_version: "encryptionkey@0"
I20260927 18:51:13.185091 19647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:13.189673 19647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:13.192916 19647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:13.193114 19647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:13.200157 19647 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:32967
I20260927 18:51:13.200220 19714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:32967 every 8 connection(s)
I20260927 18:51:13.201149 19647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:13.201370 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 19647
I20260927 18:51:13.201521 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:13.201889 30333 external_mini_cluster.cc:1428] Setting key b05d3ac86aea68719f02dbabadcb8ca7
I20260927 18:51:13.204404 19715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:13.207379 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:13.203335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35587 (local address 127.29.159.126:32967)
0927 18:51:13.203789 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:13.203803 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:13.203839 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:51:13.204697 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:13.204717 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:13.205025 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:13.205379 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:13.206198 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:13.206203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:13.206215 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:13.206225 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:13.206280 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:13.206300 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:13.206454 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:13.206523 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 18:51:13.206684 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":147,"threads_started":1}
I20260927 18:51:13.210337 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:13.210774 19715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362: Bootstrap starting.
I20260927 18:51:13.213297 19715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:13.214164 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:13.212470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45143 (local address 127.29.159.126:32967)
0927 18:51:13.212604 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:13.212608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:13.212806 (+ 198us) server_negotiation.cc:407] Connection header received
0927 18:51:13.212921 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:13.212923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:13.212953 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:13.213027 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:13.213769 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:13.213772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:13.213774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:13.213784 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:13.213807 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:13.213816 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:13.213884 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:13.213985 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 18:51:13.214025 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:51:13.214268 19715 log.cc:824] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:13.216225 19715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362: No bootstrap required, opened a new log
I20260927 18:51:13.219774 19715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfd361b3bf14ac2b2ec19dcbb2f3362" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32967 } }
I20260927 18:51:13.220009 19715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:13.220086 19715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cfd361b3bf14ac2b2ec19dcbb2f3362, State: Initialized, Role: FOLLOWER
I20260927 18:51:13.220575 19715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfd361b3bf14ac2b2ec19dcbb2f3362" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32967 } }
I20260927 18:51:13.220701 19715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:13.220785 19715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:13.220906 19715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:13.221834 19715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfd361b3bf14ac2b2ec19dcbb2f3362" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32967 } }
I20260927 18:51:13.222194 19715 leader_election.cc:302] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cfd361b3bf14ac2b2ec19dcbb2f3362; no voters:
I20260927 18:51:13.222506 19715 leader_election.cc:288] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:13.222662 19725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:13.222903 19725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [term 1 LEADER]: Becoming Leader. State: Replica: 6cfd361b3bf14ac2b2ec19dcbb2f3362, State: Running, Role: LEADER
I20260927 18:51:13.223210 19725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfd361b3bf14ac2b2ec19dcbb2f3362" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32967 } }
I20260927 18:51:13.223795 19715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:13.225065 19726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cfd361b3bf14ac2b2ec19dcbb2f3362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfd361b3bf14ac2b2ec19dcbb2f3362" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32967 } } }
I20260927 18:51:13.225179 19726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:13.225558 19735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:13.225024 19727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cfd361b3bf14ac2b2ec19dcbb2f3362. Latest consensus state: current_term: 1 leader_uuid: "6cfd361b3bf14ac2b2ec19dcbb2f3362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cfd361b3bf14ac2b2ec19dcbb2f3362" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 32967 } } }
I20260927 18:51:13.226547 19727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:13.228842 19735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:13.234432 19735 catalog_manager.cc:1347] Generated new cluster ID: 7f94ac8d7eac4e4aaf05d15b6db4c804
I20260927 18:51:13.234504 19735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:13.271162 19735 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:13.271629 19735 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:13.278810 19735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cfd361b3bf14ac2b2ec19dcbb2f3362: Generated new TSK 0
I20260927 18:51:13.279709 19735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:13.294688 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 19647
2026-09-27T18:51:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:13 dist-test-slave-8tg2 krb5kdc[19746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:13 dist-test-slave-8tg2 krb5kdc[19746](info): set up 2 sockets
Sep 27 18:51:13 dist-test-slave-8tg2 krb5kdc[19746](info): commencing operation
krb5kdc: starting...
W20260927 18:51:14.331851 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:14 dist-test-slave-8tg2 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:51:14.459231 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:43615
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:43615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:14.573199 19759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:14.573511 19759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:14.573588 19759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:14.577608 19759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:14.577669 19759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:14.577689 19759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:14.577705 19759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:14.577721 19759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:14.582895 19759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:43615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:43615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:14.583912 19759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:14.584849 19759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:14.593066 19765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:14.593082 19767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:14.593180 19759 server_base.cc:1034] running on GCE node
W20260927 18:51:14.593091 19764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:14.593900 19759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:14.595016 19759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:14.596174 19759 hybrid_clock.cc:648] HybridClock initialized: now 1790535074596160 us; error 27 us; skew 500 ppm
Sep 27 18:51:14 dist-test-slave-8tg2 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535074, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:14.606185 19759 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:51:14.607231 19759 webserver.cc:460] Webserver started at http://127.29.159.126:33707/ using document root <none> and password file <none>
I20260927 18:51:14.607904 19759 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:14.608008 19759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:14.608258 19759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:14.610443 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "50cabc54624a47d686408fee85ccf7cb"
format_stamp: "Formatted at 2026-09-27 18:51:14 on dist-test-slave-8tg2"
server_key: "277e8dfbe49b3f8020257b521984e7c5"
server_key_iv: "760741d898592268a89be97056ad7214"
server_key_version: "encryptionkey@0"
I20260927 18:51:14.611044 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "50cabc54624a47d686408fee85ccf7cb"
format_stamp: "Formatted at 2026-09-27 18:51:14 on dist-test-slave-8tg2"
server_key: "277e8dfbe49b3f8020257b521984e7c5"
server_key_iv: "760741d898592268a89be97056ad7214"
server_key_version: "encryptionkey@0"
I20260927 18:51:14.615005 19759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 18:51:14.617610 19774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:14.618924 19759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:51:14.619083 19759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "50cabc54624a47d686408fee85ccf7cb"
format_stamp: "Formatted at 2026-09-27 18:51:14 on dist-test-slave-8tg2"
server_key: "277e8dfbe49b3f8020257b521984e7c5"
server_key_iv: "760741d898592268a89be97056ad7214"
server_key_version: "encryptionkey@0"
I20260927 18:51:14.619238 19759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:14.637786 19759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:14.641171 19759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:14.641415 19759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:14.649179 19759 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:43615
I20260927 18:51:14.649190 19826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:43615 every 8 connection(s)
I20260927 18:51:14.650257 19759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:14.653297 19827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:14.656678 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 19759
I20260927 18:51:14.656822 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:14.657107 30333 external_mini_cluster.cc:1428] Setting key 0d54a7d1ceb115aa0a0f517833aecdef
I20260927 18:51:14.659514 19827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb: Bootstrap starting.
I20260927 18:51:14.662079 19827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:14.663017 19827 log.cc:824] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:14.665355 19827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb: No bootstrap required, opened a new log
Sep 27 18:51:14 dist-test-slave-8tg2 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:51:14.668944 19827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cabc54624a47d686408fee85ccf7cb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43615 } }
I20260927 18:51:14.669203 19827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:14.669296 19827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cabc54624a47d686408fee85ccf7cb, State: Initialized, Role: FOLLOWER
I20260927 18:51:14.669798 19827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cabc54624a47d686408fee85ccf7cb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43615 } }
I20260927 18:51:14.669986 19827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:14.670068 19827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:14.670194 19827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:14.671263 19827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cabc54624a47d686408fee85ccf7cb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43615 } }
I20260927 18:51:14.671691 19827 leader_election.cc:302] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50cabc54624a47d686408fee85ccf7cb; no voters:
I20260927 18:51:14.672082 19827 leader_election.cc:288] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:14.672150 19832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:14.672360 19832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [term 1 LEADER]: Becoming Leader. State: Replica: 50cabc54624a47d686408fee85ccf7cb, State: Running, Role: LEADER
I20260927 18:51:14.672659 19832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cabc54624a47d686408fee85ccf7cb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43615 } }
I20260927 18:51:14.672910 19830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:14.658516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46645 (local address 127.29.159.126:43615)
0927 18:51:14.658922 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:14.658938 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:14.658973 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:51:14.659713 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:14.659733 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:14.660082 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:14.660356 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:14.661272 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:14.662316 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:14.663539 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:14.664149 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:14.664288 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:14.667167 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:14.667173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:14.667186 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:14.667213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:14.669571 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:14.670100 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:14.670106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:14.670115 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:14.670296 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:14.670593 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:14.670595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:14.670596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:14.670990 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:14.671156 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:14.671561 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 18:51:14.671888 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":133,"threads_started":1}
I20260927 18:51:14.673405 19827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:14.673883 19832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50cabc54624a47d686408fee85ccf7cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cabc54624a47d686408fee85ccf7cb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43615 } } }
I20260927 18:51:14.675529 19834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50cabc54624a47d686408fee85ccf7cb. Latest consensus state: current_term: 1 leader_uuid: "50cabc54624a47d686408fee85ccf7cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cabc54624a47d686408fee85ccf7cb" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 43615 } } }
I20260927 18:51:14.675629 19834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:14.675632 19832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:14.677016 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 18:51:14.679145 19847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:51:14.679229 19847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:51:14.679314 19843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:14.680210 19843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:14.687244 19843 catalog_manager.cc:1347] Generated new cluster ID: bd152d79ab644336a3a6a65ca8e375fe
I20260927 18:51:14.687320 19843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:14.688123 19830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:14.679143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37527 (local address 127.29.159.126:43615)
0927 18:51:14.679256 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:14.679260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:14.679617 (+ 357us) server_negotiation.cc:407] Connection header received
0927 18:51:14.679759 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:14.679762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:14.679808 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:14.679897 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:14.680626 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:14.681180 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:14.682024 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:14.682366 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:14.682470 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:14.684234 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:14.684237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:14.684240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:14.684266 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:14.685991 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:14.686531 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:14.686534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:14.686536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:14.686628 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:14.686992 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:14.687012 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:14.687014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:14.687251 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:14.687356 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:14.687782 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 18:51:14.687913 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:51:14.706902 19843 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:14.708352 19843 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:14.719602 19843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50cabc54624a47d686408fee85ccf7cb: Generated new TSK 0
I20260927 18:51:14.720678 19843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:14.733341 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 19759
2026-09-27T18:51:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:14 dist-test-slave-8tg2 krb5kdc[19858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:14 dist-test-slave-8tg2 krb5kdc[19858](info): set up 2 sockets
Sep 27 18:51:14 dist-test-slave-8tg2 krb5kdc[19858](info): commencing operation
krb5kdc: starting...
W20260927 18:51:15.775866 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:15 dist-test-slave-8tg2 krb5kdc[19858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:51:15.898643 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:40817
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:40817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:16.011196 19871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:16.011551 19871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:16.011663 19871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:16.015729 19871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:16.015816 19871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:16.015852 19871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:16.015913 19871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:16.015949 19871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:16.021229 19871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:40817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:40817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:16.022356 19871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:16.023464 19871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:16.032076 19879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:16.032177 19871 server_base.cc:1034] running on GCE node
W20260927 18:51:16.032131 19877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:16.032541 19876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:16.033160 19871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:16.034329 19871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:16.035527 19871 hybrid_clock.cc:648] HybridClock initialized: now 1790535076035504 us; error 25 us; skew 500 ppm
Sep 27 18:51:16 dist-test-slave-8tg2 krb5kdc[19858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535076, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:16.038516 19871 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:51:16.039379 19871 webserver.cc:460] Webserver started at http://127.29.159.126:39707/ using document root <none> and password file <none>
I20260927 18:51:16.039986 19871 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:16.040050 19871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:16.040225 19871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:16.042332 19871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "3f78f0d4982543d2bb328510fca68b0c"
format_stamp: "Formatted at 2026-09-27 18:51:16 on dist-test-slave-8tg2"
server_key: "b8354be8da8bd23ade188a1be90a88d7"
server_key_iv: "2662e0c7f54096aaea6247e48735b7d2"
server_key_version: "encryptionkey@0"
I20260927 18:51:16.042872 19871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "3f78f0d4982543d2bb328510fca68b0c"
format_stamp: "Formatted at 2026-09-27 18:51:16 on dist-test-slave-8tg2"
server_key: "b8354be8da8bd23ade188a1be90a88d7"
server_key_iv: "2662e0c7f54096aaea6247e48735b7d2"
server_key_version: "encryptionkey@0"
I20260927 18:51:16.046622 19871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 18:51:16.049110 19886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:16.050319 19871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:51:16.050514 19871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "3f78f0d4982543d2bb328510fca68b0c"
format_stamp: "Formatted at 2026-09-27 18:51:16 on dist-test-slave-8tg2"
server_key: "b8354be8da8bd23ade188a1be90a88d7"
server_key_iv: "2662e0c7f54096aaea6247e48735b7d2"
server_key_version: "encryptionkey@0"
I20260927 18:51:16.050623 19871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:16.061903 19871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:16.064939 19871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:16.065140 19871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:16.072214 19871 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:40817
I20260927 18:51:16.072242 19938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:40817 every 8 connection(s)
I20260927 18:51:16.073197 19871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:16.074983 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 19871
I20260927 18:51:16.075117 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:16.075412 30333 external_mini_cluster.cc:1428] Setting key 921f61c2f0a1f810f432a031c320a2fd
I20260927 18:51:16.076399 19939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:16.083570 19939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c: Bootstrap starting.
I20260927 18:51:16.085847 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:16.076947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47071 (local address 127.29.159.126:40817)
0927 18:51:16.077378 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:16.077398 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:16.077442 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:51:16.078295 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:16.078319 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:16.078780 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:16.079222 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:16.079688 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:16.081209 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:16.082220 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:16.082928 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:16.083102 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:16.084070 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:16.084075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:16.084089 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:16.084490 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:16.084546 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:16.084569 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:16.084651 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:16.084831 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 18:51:16.085085 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260927 18:51:16.086148 19939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:16.087021 19939 log.cc:824] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:16.088932 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:16.089305 19939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c: No bootstrap required, opened a new log
I20260927 18:51:16.093376 19939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f78f0d4982543d2bb328510fca68b0c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40817 } }
I20260927 18:51:16.093667 19939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:16.093729 19939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f78f0d4982543d2bb328510fca68b0c, State: Initialized, Role: FOLLOWER
I20260927 18:51:16.094213 19939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f78f0d4982543d2bb328510fca68b0c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40817 } }
I20260927 18:51:16.094370 19939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:16.094451 19939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:16.094566 19939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:16.095285 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:16.091003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41221 (local address 127.29.159.126:40817)
0927 18:51:16.091136 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:16.091142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:16.091354 (+ 212us) server_negotiation.cc:407] Connection header received
0927 18:51:16.091543 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:16.091547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:16.091608 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:16.091705 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:16.091949 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:16.092752 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:16.093562 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:16.093917 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:16.094029 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:16.094883 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:16.094887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:16.094889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:16.094902 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:16.094928 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:16.094940 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:16.095027 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:16.095094 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 18:51:16.095147 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:51:16.095765 19939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f78f0d4982543d2bb328510fca68b0c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40817 } }
I20260927 18:51:16.096124 19939 leader_election.cc:302] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f78f0d4982543d2bb328510fca68b0c; no voters:
I20260927 18:51:16.096483 19939 leader_election.cc:288] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:16.096601 19949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:16.096812 19949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [term 1 LEADER]: Becoming Leader. State: Replica: 3f78f0d4982543d2bb328510fca68b0c, State: Running, Role: LEADER
I20260927 18:51:16.097157 19949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f78f0d4982543d2bb328510fca68b0c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40817 } }
I20260927 18:51:16.097831 19939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:16.099013 19950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f78f0d4982543d2bb328510fca68b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f78f0d4982543d2bb328510fca68b0c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40817 } } }
I20260927 18:51:16.099023 19951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f78f0d4982543d2bb328510fca68b0c. Latest consensus state: current_term: 1 leader_uuid: "3f78f0d4982543d2bb328510fca68b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f78f0d4982543d2bb328510fca68b0c" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 40817 } } }
I20260927 18:51:16.099205 19950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:16.099226 19951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:16.099915 19958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:16.102820 19958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:16.108577 19958 catalog_manager.cc:1347] Generated new cluster ID: 844aca6cb2b3448691d8e3afd02291c7
I20260927 18:51:16.108670 19958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:16.121250 19958 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:16.122660 19958 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:16.137223 19958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f78f0d4982543d2bb328510fca68b0c: Generated new TSK 0
I20260927 18:51:16.138311 19958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:16.149397 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 19871
2026-09-27T18:51:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:16 dist-test-slave-8tg2 krb5kdc[19970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:16 dist-test-slave-8tg2 krb5kdc[19970](info): set up 2 sockets
Sep 27 18:51:16 dist-test-slave-8tg2 krb5kdc[19970](info): commencing operation
krb5kdc: starting...
W20260927 18:51:17.199885 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:17 dist-test-slave-8tg2 krb5kdc[19970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:51:17.327375 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:38723
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:38723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:17.449052 19983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:17.449435 19983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:17.449548 19983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:17.453990 19983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:17.454082 19983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:17.454119 19983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:17.454172 19983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:17.454221 19983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:17.460175 19983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:38723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:38723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:17.461406 19983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:17.462539 19983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:17.471275 19988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:17.471277 19991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:17.471318 19983 server_base.cc:1034] running on GCE node
W20260927 18:51:17.471275 19989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:17.472129 19983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:17.473278 19983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:17.474502 19983 hybrid_clock.cc:648] HybridClock initialized: now 1790535077474478 us; error 38 us; skew 500 ppm
Sep 27 18:51:17 dist-test-slave-8tg2 krb5kdc[19970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535077, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:17.477325 19983 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:51:17.478387 19983 webserver.cc:460] Webserver started at http://127.29.159.126:33847/ using document root <none> and password file <none>
I20260927 18:51:17.478997 19983 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:17.479061 19983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:17.479300 19983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:17.481564 19983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "bde7cb5f9c4b4d6785ff26a544506a89"
format_stamp: "Formatted at 2026-09-27 18:51:17 on dist-test-slave-8tg2"
server_key: "f9ff6e727ae76a8d8ab71e7b2a194883"
server_key_iv: "26e8a4c8533af9cd18559bfec68b1b83"
server_key_version: "encryptionkey@0"
I20260927 18:51:17.482147 19983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "bde7cb5f9c4b4d6785ff26a544506a89"
format_stamp: "Formatted at 2026-09-27 18:51:17 on dist-test-slave-8tg2"
server_key: "f9ff6e727ae76a8d8ab71e7b2a194883"
server_key_iv: "26e8a4c8533af9cd18559bfec68b1b83"
server_key_version: "encryptionkey@0"
I20260927 18:51:17.486155 19983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:51:17.488917 19998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:17.490276 19983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:51:17.490437 19983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "bde7cb5f9c4b4d6785ff26a544506a89"
format_stamp: "Formatted at 2026-09-27 18:51:17 on dist-test-slave-8tg2"
server_key: "f9ff6e727ae76a8d8ab71e7b2a194883"
server_key_iv: "26e8a4c8533af9cd18559bfec68b1b83"
server_key_version: "encryptionkey@0"
I20260927 18:51:17.490573 19983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:17.495584 19983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:17.499019 19983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:17.499228 19983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:17.506400 19983 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:38723
I20260927 18:51:17.506415 20050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:38723 every 8 connection(s)
I20260927 18:51:17.507364 19983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:17.510171 20051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:17.514175 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 19983
I20260927 18:51:17.514335 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:17.514614 30333 external_mini_cluster.cc:1428] Setting key d3d5445850cd40a7a09d3451003362a9
I20260927 18:51:17.516028 20051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89: Bootstrap starting.
I20260927 18:51:17.518414 20051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:17.519282 20051 log.cc:824] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:17.520030 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:17.516067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49973 (local address 127.29.159.126:38723)
0927 18:51:17.516509 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:17.516525 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:17.516561 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:51:17.517206 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:17.517223 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:17.517525 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:17.517813 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:17.518653 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:17.518658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:17.518670 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:17.518680 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:17.518733 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:17.518752 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:17.518836 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:17.518965 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 18:51:17.519131 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":122,"threads_started":1}
I20260927 18:51:17.521467 20051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89: No bootstrap required, opened a new log
I20260927 18:51:17.522782 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:17.525475 20051 raft_consensus.cc:348] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7cb5f9c4b4d6785ff26a544506a89" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38723 } }
I20260927 18:51:17.525779 20051 raft_consensus.cc:374] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:51:17.525869 20051 raft_consensus.cc:729] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde7cb5f9c4b4d6785ff26a544506a89, State: Initialized, Role: FOLLOWER
I20260927 18:51:17.526391 20051 consensus_queue.cc:260] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7cb5f9c4b4d6785ff26a544506a89" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38723 } }
I20260927 18:51:17.526556 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:17.524820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39581 (local address 127.29.159.126:38723)
0927 18:51:17.524951 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:17.524955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:17.525168 (+ 213us) server_negotiation.cc:407] Connection header received
0927 18:51:17.525280 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:17.525282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:17.525310 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:17.525374 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:17.526142 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:17.526145 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:17.526149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:51:17.526165 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:17.526196 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:51:17.526206 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:17.526274 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:17.526370 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 18:51:17.526400 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:51:17.526584 20051 raft_consensus.cc:388] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:51:17.526648 20051 raft_consensus.cc:482] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:51:17.526758 20051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:17.527851 20051 raft_consensus.cc:504] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7cb5f9c4b4d6785ff26a544506a89" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38723 } }
I20260927 18:51:17.528193 20051 leader_election.cc:302] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bde7cb5f9c4b4d6785ff26a544506a89; no voters:
I20260927 18:51:17.528479 20051 leader_election.cc:288] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:51:17.528606 20061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:17.528851 20061 raft_consensus.cc:686] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [term 1 LEADER]: Becoming Leader. State: Replica: bde7cb5f9c4b4d6785ff26a544506a89, State: Running, Role: LEADER
I20260927 18:51:17.529189 20061 consensus_queue.cc:237] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7cb5f9c4b4d6785ff26a544506a89" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38723 } }
I20260927 18:51:17.529678 20051 sys_catalog.cc:564] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:17.530849 20063 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde7cb5f9c4b4d6785ff26a544506a89. Latest consensus state: current_term: 1 leader_uuid: "bde7cb5f9c4b4d6785ff26a544506a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7cb5f9c4b4d6785ff26a544506a89" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38723 } } }
I20260927 18:51:17.530978 20063 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:17.531270 20062 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bde7cb5f9c4b4d6785ff26a544506a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7cb5f9c4b4d6785ff26a544506a89" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 38723 } } }
I20260927 18:51:17.531359 20062 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:17.531345 20070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:17.534621 20070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:17.540326 20070 catalog_manager.cc:1347] Generated new cluster ID: 4ae3b2bc9c564a9ebb2b32c2703be808
I20260927 18:51:17.540405 20070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:17.551038 20070 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:17.551486 20070 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:17.556633 20070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bde7cb5f9c4b4d6785ff26a544506a89: Generated new TSK 0
I20260927 18:51:17.557444 20070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:51:17.571709 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 19983
2026-09-27T18:51:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1422 ms)
[----------] 18 tests from AuthTokenIssuingTest (25711 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:17 dist-test-slave-8tg2 krb5kdc[20082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:17 dist-test-slave-8tg2 krb5kdc[20082](info): set up 2 sockets
Sep 27 18:51:17 dist-test-slave-8tg2 krb5kdc[20082](info): commencing operation
krb5kdc: starting...
W20260927 18:51:18.607903 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:18 dist-test-slave-8tg2 krb5kdc[20082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:51:18.734648 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:46043
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:46043,127.29.159.125:44369,127.29.159.124:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:18.846581 20095 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:51:18.846951 20095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:18.847014 20095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:18.847093 20095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:18.851261 20095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:18.851367 20095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:18.851404 20095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:18.851480 20095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:18.851528 20095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:18.857072 20095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:46043,127.29.159.125:44369,127.29.159.124:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:46043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:18.858239 20095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:18.859361 20095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:18.868096 20100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:18.868096 20103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:18.868110 20095 server_base.cc:1034] running on GCE node
W20260927 18:51:18.868104 20101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:18.869022 20095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:18.870182 20095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:18.871377 20095 hybrid_clock.cc:648] HybridClock initialized: now 1790535078871364 us; error 35 us; skew 500 ppm
Sep 27 18:51:18 dist-test-slave-8tg2 krb5kdc[20082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535078, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:18.874171 20095 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:51:18.875077 20095 webserver.cc:460] Webserver started at http://127.29.159.126:40575/ using document root <none> and password file <none>
I20260927 18:51:18.875699 20095 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:18.875765 20095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:18.875981 20095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:18.878054 20095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "5460791c78b34e51b9697ec981919160"
format_stamp: "Formatted at 2026-09-27 18:51:18 on dist-test-slave-8tg2"
server_key: "53abc716056477b870e430b2ab7cf852"
server_key_iv: "7a6de5bc2d582b708e62739ee7211159"
server_key_version: "encryptionkey@0"
I20260927 18:51:18.878618 20095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "5460791c78b34e51b9697ec981919160"
format_stamp: "Formatted at 2026-09-27 18:51:18 on dist-test-slave-8tg2"
server_key: "53abc716056477b870e430b2ab7cf852"
server_key_iv: "7a6de5bc2d582b708e62739ee7211159"
server_key_version: "encryptionkey@0"
I20260927 18:51:18.882269 20095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:51:18.884846 20110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:18.886137 20095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:51:18.886267 20095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "5460791c78b34e51b9697ec981919160"
format_stamp: "Formatted at 2026-09-27 18:51:18 on dist-test-slave-8tg2"
server_key: "53abc716056477b870e430b2ab7cf852"
server_key_iv: "7a6de5bc2d582b708e62739ee7211159"
server_key_version: "encryptionkey@0"
I20260927 18:51:18.886375 20095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:18.899856 20095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:18.902835 20095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:18.903045 20095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:18.910053 20095 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:46043
I20260927 18:51:18.910318 20162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:46043 every 8 connection(s)
I20260927 18:51:18.911094 20095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:18.913993 20163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:18.917856 20163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:51:18.921406 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 20095
I20260927 18:51:18.921535 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:18.921821 30333 external_mini_cluster.cc:1428] Setting key 7981ed3c2f4e5d925ace1a988156d278
Sep 27 18:51:18 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535078, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.159.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.125@KRBTEST.COM" created.
I20260927 18:51:18.934165 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:18.920931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.29.159.126:46043)
0927 18:51:18.921201 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:18.921210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:18.921245 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:51:18.922193 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:18.922200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:18.922506 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:18.922706 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:18.923503 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:18.924404 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:18.925376 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:18.925982 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:18.926173 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:18.928912 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:18.928921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:18.928939 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:18.928972 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:18.931101 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:18.931667 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:18.931675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:18.931676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:18.931788 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:18.932143 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:18.932145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:18.932146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:18.932611 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:18.932720 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:18.933195 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 18:51:18.933360 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
I20260927 18:51:18.934171 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:18.920198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46043 (local address 127.0.0.1:60198)
0927 18:51:18.920683 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:18.920726 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:18.921762 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:18.922721 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:18.922725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:18.923055 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:18.923325 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:18.923337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:18.924529 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:18.924532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:18.925262 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:18.925270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:18.926168 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:18.926172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:18.926281 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:18.927030 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:18.927048 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:18.928743 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:18.931208 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:18.931216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:18.931226 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:18.931556 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:18.931898 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:18.931901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:18.931902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:18.932024 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:18.932736 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:18.932742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:18.932874 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 18:51:18.933024 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 18:51:18.933291 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
I20260927 18:51:18.937932 20163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:51:18.938773 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:18.938508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44369 (local address 127.0.0.1:45604)
0927 18:51:18.938634 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:18.938710 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:51:18.939029 20114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44369: connect: Connection refused (error 111)
I20260927 18:51:18.939839 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:18.939639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44369 (local address 127.0.0.1:45610)
0927 18:51:18.939740 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:18.939789 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 18:51:18.939978 20163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.125:44369: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44369: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab.
Entry for principal kudu/127.29.159.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab.
I20260927 18:51:18.951162 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.125:44369
--webserver_interface=127.29.159.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab
--principal=kudu/127.29.159.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:46043,127.29.159.125:44369,127.29.159.124:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:51:18.989228 20163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } attempt: 1
I20260927 18:51:18.990149 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:18.989802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44369 (local address 127.0.0.1:45626)
0927 18:51:18.990008 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:18.990070 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 18:51:18.990859 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:18.990643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44369 (local address 127.0.0.1:45638)
0927 18:51:18.990743 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:18.990797 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 18:51:18.991006 20163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.125:44369: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44369: connect: Connection refused (error 111)
I20260927 18:51:19.059196 20163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } attempt: 2
I20260927 18:51:19.060235 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.059856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44369 (local address 127.0.0.1:45642)
0927 18:51:19.060056 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.060135 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 18:51:19.061131 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.060883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44369 (local address 127.0.0.1:45650)
0927 18:51:19.061018 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.061072 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 18:51:19.061403 20163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.125:44369: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44369: connect: Connection refused (error 111)
W20260927 18:51:19.068897 20170 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:51:19.069218 20170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:19.069293 20170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:19.069363 20170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:19.073730 20170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:19.073801 20170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:19.073822 20170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:19.073838 20170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:19.073874 20170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:19.080265 20170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:46043,127.29.159.125:44369,127.29.159.124:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab
--principal=kudu/127.29.159.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.125:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.159.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:19.081467 20170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:19.082540 20170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:19.091109 20175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:19.091111 20176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:19.091111 20178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:19.091480 20170 server_base.cc:1034] running on GCE node
I20260927 18:51:19.092041 20170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:19.093129 20170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:19.094295 20170 hybrid_clock.cc:648] HybridClock initialized: now 1790535079094270 us; error 39 us; skew 500 ppm
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535079, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:19.097112 20170 init.cc:377] Logged in from keytab as kudu/127.29.159.125@KRBTEST.COM (short username kudu)
I20260927 18:51:19.098002 20170 webserver.cc:460] Webserver started at http://127.29.159.125:46413/ using document root <none> and password file <none>
I20260927 18:51:19.098564 20170 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:19.098652 20170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:19.098873 20170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:19.100951 20170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/data/instance:
uuid: "229da1707d034573949dc89cba9f62a9"
format_stamp: "Formatted at 2026-09-27 18:51:19 on dist-test-slave-8tg2"
server_key: "8142b2f0b4ab92e4dca9a05fc67e9bc2"
server_key_iv: "4dec414e2901f0e82911d2637ce7b23b"
server_key_version: "encryptionkey@0"
I20260927 18:51:19.101526 20170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/wal/instance:
uuid: "229da1707d034573949dc89cba9f62a9"
format_stamp: "Formatted at 2026-09-27 18:51:19 on dist-test-slave-8tg2"
server_key: "8142b2f0b4ab92e4dca9a05fc67e9bc2"
server_key_iv: "4dec414e2901f0e82911d2637ce7b23b"
server_key_version: "encryptionkey@0"
I20260927 18:51:19.105239 20170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:51:19.107640 20185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:19.108896 20170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:51:19.108994 20170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/wal
uuid: "229da1707d034573949dc89cba9f62a9"
format_stamp: "Formatted at 2026-09-27 18:51:19 on dist-test-slave-8tg2"
server_key: "8142b2f0b4ab92e4dca9a05fc67e9bc2"
server_key_iv: "4dec414e2901f0e82911d2637ce7b23b"
server_key_version: "encryptionkey@0"
I20260927 18:51:19.109089 20170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:19.124256 20170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:19.127079 20170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:19.127230 20170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:19.134287 20170 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.125:44369
I20260927 18:51:19.134584 20237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.125:44369 every 8 connection(s)
I20260927 18:51:19.135298 20170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
I20260927 18:51:19.138046 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 20170
I20260927 18:51:19.138170 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-1/wal/instance
I20260927 18:51:19.138266 20238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:19.138489 30333 external_mini_cluster.cc:1428] Setting key ab6898da9e81b8cef6838a75ec54b1e8
I20260927 18:51:19.142521 20238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.159.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.124@KRBTEST.COM" created.
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535079, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:51:19.152559 20163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } attempt: 3
I20260927 18:51:19.158073 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.144433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.29.159.126:46043)
0927 18:51:19.144609 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.144616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.145398 (+ 782us) server_negotiation.cc:407] Connection header received
0927 18:51:19.146377 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.146380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.146431 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.146521 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.147790 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.148383 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.149956 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.150189 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.150306 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.152896 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.152904 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.152906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.152937 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.155667 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.156310 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.156316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.156319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.156412 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.156798 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.156803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.156805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.156989 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.157092 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.157788 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.157949 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:51:19.159111 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.144764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46043 (local address 127.0.0.1:60204)
0927 18:51:19.145222 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.145273 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:19.146160 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:19.146670 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:19.146676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:19.147079 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:19.147646 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.147661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.148498 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.148501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.149821 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.149835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.150350 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.150357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.150569 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.151163 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:19.151180 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:19.152735 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:19.155809 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.155814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.155831 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.156186 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.156513 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.156516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.156518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.156709 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.157102 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:19.157110 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:19.157398 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 18:51:19.157630 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 18:51:19.158066 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":129,"threads_started":1}
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535078, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
I20260927 18:51:19.160813 20238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:51:19.163739 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.153072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44369 (local address 127.0.0.1:45660)
0927 18:51:19.153212 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.153231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:19.153310 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:19.154177 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:19.154182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:19.154517 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:19.154659 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.154668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.155766 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.155770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.156548 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.156561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.157347 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.157350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.157455 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.157934 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:19.157948 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:19.159938 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:19.162160 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.162162 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.162164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.162418 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.162680 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.162682 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.162684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.162777 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.163203 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:19.163206 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:19.163307 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 18:51:19.163445 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 18:51:19.163564 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 18:51:19.163915 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.153316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.29.159.125:44369)
0927 18:51:19.153658 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.153671 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.153695 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:51:19.153741 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.153749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.154000 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.154176 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.154815 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.155645 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.156669 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.157239 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.157346 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.160075 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.160086 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.160103 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.160134 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.162015 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.162520 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.162524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.162525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.162594 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.162861 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.162864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.162865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.163117 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.163214 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.163593 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.163732 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":213,"spinlock_wait_cycles":1792,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab.
Entry for principal kudu/127.29.159.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab.
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535079, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
I20260927 18:51:19.166574 20163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:51:19.167237 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.124:37331
--webserver_interface=127.29.159.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab
--principal=kudu/127.29.159.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:46043,127.29.159.125:44369,127.29.159.124:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:51:19.167629 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.167023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53820)
0927 18:51:19.167169 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.167544 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 18:51:19.168444 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.168103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53822)
0927 18:51:19.168292 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.168354 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 18:51:19.168625 20163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:37331: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
I20260927 18:51:19.170138 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.161370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44369 (local address 127.0.0.1:45670)
0927 18:51:19.161499 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.161513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:19.161587 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:19.162041 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:19.162047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:19.162216 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:19.162303 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.162307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.163416 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.163420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.164284 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.164293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.164943 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.164946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.165019 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.165419 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:19.165432 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:19.166786 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:19.168746 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.168753 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.168755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.168944 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.169218 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.169221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.169223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.169299 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.169651 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:19.169655 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:19.169737 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 18:51:19.169833 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 18:51:19.169975 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 18:51:19.170188 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.161489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.29.159.125:44369)
0927 18:51:19.161817 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.161828 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.161844 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:51:19.161880 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.161883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.161935 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.162004 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.162428 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.163294 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.164395 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.164819 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.164930 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.166935 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.166939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.166941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.166963 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.168656 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.169036 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.169041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.169043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.169128 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.169390 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.169393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.169394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.169557 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.169651 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.169885 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.170033 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":252,"thread_start_us":146,"threads_started":1}
I20260927 18:51:19.170794 20238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:51:19.171499 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.171223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53836)
0927 18:51:19.171326 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.171402 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 18:51:19.171720 20186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
I20260927 18:51:19.172551 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.172353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53842)
0927 18:51:19.172454 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.172497 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 18:51:19.172700 20238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:37331: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
I20260927 18:51:19.199790 20163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } attempt: 1
I20260927 18:51:19.200680 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.200369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53848)
0927 18:51:19.200537 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.200601 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 18:51:19.201450 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.201210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53852)
0927 18:51:19.201322 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.201372 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 18:51:19.201596 20163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:37331: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
I20260927 18:51:19.221940 20238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } attempt: 1
I20260927 18:51:19.222916 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.222624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53868)
0927 18:51:19.222776 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.222838 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 18:51:19.223677 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.223414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53870)
0927 18:51:19.223578 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.223622 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 18:51:19.223802 20238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:37331: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
I20260927 18:51:19.276741 20163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } attempt: 2
I20260927 18:51:19.277559 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.277264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53878)
0927 18:51:19.277405 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.277473 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 18:51:19.278273 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.278072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53892)
0927 18:51:19.278178 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.278224 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 18:51:19.278412 20163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:37331: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
W20260927 18:51:19.278873 20245 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:51:19.279192 20245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:19.279250 20245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:19.279331 20245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:19.283356 20245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:19.283460 20245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:19.283514 20245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:19.283546 20245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:19.283578 20245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:19.289008 20245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:46043,127.29.159.125:44369,127.29.159.124:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab
--principal=kudu/127.29.159.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.124:37331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.159.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:19.290138 20245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:19.291217 20245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:51:19.291944 20238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } attempt: 2
I20260927 18:51:19.292649 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.292376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53902)
0927 18:51:19.292523 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.292582 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 18:51:19.293377 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.293174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53908)
0927 18:51:19.293276 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.293325 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 18:51:19.293493 20238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:37331: Network error: Client connection negotiation failed: client connection to 127.29.159.124:37331: connect: Connection refused (error 111)
W20260927 18:51:19.299685 20254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:19.299675 20251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:19.299661 20252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:19.299793 20245 server_base.cc:1034] running on GCE node
I20260927 18:51:19.300537 20245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:19.301585 20245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:19.302766 20245 hybrid_clock.cc:648] HybridClock initialized: now 1790535079302735 us; error 46 us; skew 500 ppm
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535079, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:19.305580 20245 init.cc:377] Logged in from keytab as kudu/127.29.159.124@KRBTEST.COM (short username kudu)
I20260927 18:51:19.306494 20245 webserver.cc:460] Webserver started at http://127.29.159.124:45831/ using document root <none> and password file <none>
I20260927 18:51:19.307054 20245 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:19.307142 20245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:19.307363 20245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:19.309398 20245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/data/instance:
uuid: "50f2d7d23ef7444ca8271d72e311199a"
format_stamp: "Formatted at 2026-09-27 18:51:19 on dist-test-slave-8tg2"
server_key: "1d4e934723547d39a83a5da171b40470"
server_key_iv: "495f091b32b6522e9aff40ecc498965e"
server_key_version: "encryptionkey@0"
I20260927 18:51:19.309973 20245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/wal/instance:
uuid: "50f2d7d23ef7444ca8271d72e311199a"
format_stamp: "Formatted at 2026-09-27 18:51:19 on dist-test-slave-8tg2"
server_key: "1d4e934723547d39a83a5da171b40470"
server_key_iv: "495f091b32b6522e9aff40ecc498965e"
server_key_version: "encryptionkey@0"
I20260927 18:51:19.313822 20245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:51:19.316316 20261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:19.317668 20245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:51:19.317840 20245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/wal
uuid: "50f2d7d23ef7444ca8271d72e311199a"
format_stamp: "Formatted at 2026-09-27 18:51:19 on dist-test-slave-8tg2"
server_key: "1d4e934723547d39a83a5da171b40470"
server_key_iv: "495f091b32b6522e9aff40ecc498965e"
server_key_version: "encryptionkey@0"
I20260927 18:51:19.317965 20245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:19.346813 20245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:19.350129 20245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:19.350339 20245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:19.357463 20245 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.124:37331
I20260927 18:51:19.357764 20313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.124:37331 every 8 connection(s)
I20260927 18:51:19.358675 20245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
I20260927 18:51:19.361756 20314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:19.364076 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 20245
I20260927 18:51:19.364197 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534801142164-30333-0/minicluster-data/master-2/wal/instance
I20260927 18:51:19.364464 30333 external_mini_cluster.cc:1428] Setting key 3764b96d097e57138210778b5b9e2e5a
I20260927 18:51:19.366163 20314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535079, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
I20260927 18:51:19.377564 20163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } attempt: 3
I20260927 18:51:19.378099 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.367860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.29.159.126:46043)
0927 18:51:19.368327 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.368331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.368711 (+ 380us) server_negotiation.cc:407] Connection header received
0927 18:51:19.368797 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.368800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.368851 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.368937 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.369611 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.370201 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.371300 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.371679 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.371785 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.374183 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.374189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.374195 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.374231 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.375809 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.376331 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.376334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.376335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.376401 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.376755 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.376758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.376760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.376983 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.377069 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.377826 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.377945 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373}
I20260927 18:51:19.379317 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.367807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46043 (local address 127.0.0.1:60208)
0927 18:51:19.368572 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.368620 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:19.368726 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:19.368993 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:19.368997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:19.369305 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:19.369481 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.369491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.370319 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.370323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.371195 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.371204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.371834 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.371841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.371954 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.372322 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:19.372342 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:19.373958 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:19.375917 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.375926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.375938 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.376232 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.376485 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.376488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.376490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.376667 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.377083 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:19.377089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:19.377384 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 18:51:19.377592 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 18:51:19.377926 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260927 18:51:19.379315 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.365877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.29.159.124:37331)
0927 18:51:19.366308 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.366323 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.366359 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:51:19.367036 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.367052 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.367379 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.367761 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.368599 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.369776 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.370852 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.371421 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.371625 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.374584 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.374592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.374607 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.374637 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.376548 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.376991 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.376995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.376996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.377088 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.377339 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.377342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.377344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.377681 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.377849 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.378714 (+ 865us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.378845 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":293,"spinlock_wait_cycles":32512,"thread_start_us":137,"threads_started":1}
I20260927 18:51:19.381810 20314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535078, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
I20260927 18:51:19.384652 20238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } attempt: 3
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:51:19.388813 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.377942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53924)
0927 18:51:19.378215 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.378232 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:19.378302 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:19.379168 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:19.379171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:19.379349 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:19.379492 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.379504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.380602 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.380605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.381207 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.381215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.381860 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.381864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.381953 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.382401 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:19.382418 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:19.384216 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:19.387118 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.387121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.387124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.387376 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.387759 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.387762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.387764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.387858 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.388302 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:19.388305 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:19.388396 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 18:51:19.388494 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:51:19.388639 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535079, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
I20260927 18:51:19.389523 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.378189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.29.159.124:37331)
0927 18:51:19.378943 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.378952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.378964 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:51:19.378993 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.378995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.379029 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.379142 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.379660 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.380470 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.381334 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.381713 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.382186 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.384345 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.384349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.384352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.384378 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.386989 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.387535 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.387538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.387540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.387649 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.387979 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.387983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.387985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.388188 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.389092 (+ 904us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.389179 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.389297 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":148,"threads_started":1}
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535079, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
I20260927 18:51:19.393854 20163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160: Bootstrap starting.
I20260927 18:51:19.394104 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.382247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.29.159.126:46043)
0927 18:51:19.382368 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.382372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.382762 (+ 390us) server_negotiation.cc:407] Connection header received
0927 18:51:19.382871 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.382874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.382919 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.383001 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.383969 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.384720 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.385852 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.386196 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.386276 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.390265 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.390269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.390272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.390293 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.392077 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.392604 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.392610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.392615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.392706 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.393055 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.393059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.393061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.393262 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.393445 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.393833 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.393957 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:51:19.394927 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.385067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53938)
0927 18:51:19.385349 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.385365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:19.385452 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:19.386047 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:19.386050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:19.386257 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:19.386383 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.386394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.387610 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.387613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.388512 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.388521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.389205 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.389209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.389306 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.389773 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:19.389788 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:19.391147 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:19.393374 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.393378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.393380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.393628 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.393915 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.393918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.393920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.394002 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.394440 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:19.394443 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:19.394540 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 18:51:19.394638 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:51:19.394755 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260927 18:51:19.395713 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.382565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.29.159.125:44369)
0927 18:51:19.382709 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.382715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.383133 (+ 418us) server_negotiation.cc:407] Connection header received
0927 18:51:19.383254 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.383257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.383314 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.383408 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.384032 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.385159 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.386468 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.387124 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.387213 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.390238 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.390242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.390244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.390269 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.393164 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.393640 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.393646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.393651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.393729 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.394100 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.394102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.394104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.394378 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.394446 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.395403 (+ 957us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.395553 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:51:19.396139 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.385617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.29.159.124:37331)
0927 18:51:19.385776 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.385781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.385797 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:51:19.385835 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.385839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.385880 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.386515 (+ 635us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.386605 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.387478 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.388656 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.389050 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.389508 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.391292 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.391297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.391301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.391328 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.393262 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.393737 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.393740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.393741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.393815 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.394107 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.394110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.394112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.394293 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.395416 (+ 1123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.395523 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.395723 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:51:19.396688 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.382286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44369 (local address 127.0.0.1:45676)
0927 18:51:19.383058 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.383074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:19.383169 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:19.383491 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:19.383496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:19.383667 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:19.383897 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.383909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.385285 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.385288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.386339 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.386352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.387380 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.387385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.387532 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.388029 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:19.388049 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:19.390079 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:19.393288 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.393294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.393299 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.393530 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.393833 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.393839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.393844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.393934 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.394876 (+ 942us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:19.394879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:19.395210 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 18:51:19.396007 (+ 797us) client_negotiation.cc:240] Negotiation successful
0927 18:51:19.396140 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"spinlock_wait_cycles":768}
I20260927 18:51:19.396963 20163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:19.397545 20314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:51:19.397997 20163 log.cc:824] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:19.399197 20238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9: Bootstrap starting.
I20260927 18:51:19.400240 20163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160: No bootstrap required, opened a new log
I20260927 18:51:19.401751 20238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9: Neither blocks nor log segments found. Creating new log.
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
I20260927 18:51:19.402617 20238 log.cc:824] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:51:19 dist-test-slave-8tg2 krb5kdc[20082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535079, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
I20260927 18:51:19.405417 20163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } }
I20260927 18:51:19.405741 20163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5460791c78b34e51b9697ec981919160, State: Initialized, Role: FOLLOWER
I20260927 18:51:19.406148 20163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } }
I20260927 18:51:19.406684 20323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } } }
I20260927 18:51:19.406894 20323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:19.407052 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.396410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.29.159.125:44369)
0927 18:51:19.396515 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.396521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.396539 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:51:19.396651 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.396655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.396697 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.396752 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.397693 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.398470 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.399681 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.400014 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.400094 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.403200 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.403204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.403207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.403233 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.404994 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.405526 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.405529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.405531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.405635 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.405986 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.405989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.405991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.406219 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.406383 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.406791 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.406906 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 18:51:19.407738 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.398004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:37331 (local address 127.0.0.1:53946)
0927 18:51:19.398563 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:19.398576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:19.398649 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:19.398922 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:19.398924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:19.399077 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:19.399150 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.399156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.400151 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.400154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.401253 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:19.401264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.402072 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.402075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:19.402144 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.402578 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:19.402663 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:19.404256 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:19.406039 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.406045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.406048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.406230 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.406589 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:19.406591 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:19.406593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:19.406675 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:19.407213 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:19.407215 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:19.407288 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 18:51:19.407440 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 18:51:19.407568 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":118,"threads_started":1}
I20260927 18:51:19.407739 20163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:19.407742 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.398310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.29.159.124:37331)
0927 18:51:19.398477 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.398481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.398717 (+ 236us) server_negotiation.cc:407] Connection header received
0927 18:51:19.398741 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.398743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.398773 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.398829 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.399323 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.400035 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.401376 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.401756 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.401953 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.404371 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.404374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.404377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.404398 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.405942 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.406395 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.406397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.406399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.406492 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.406771 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.406773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.406775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.407055 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.407175 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.407487 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.407630 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":106}
I20260927 18:51:19.410769 20314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a: Bootstrap starting.
I20260927 18:51:19.410902 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:19.412406 20137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5460791c78b34e51b9697ec981919160"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60218
I20260927 18:51:19.412650 20137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:51:19.412760 20238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9: No bootstrap required, opened a new log
I20260927 18:51:19.412772 20137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [term 0 FOLLOWER]: Advancing to term 1
W20260927 18:51:19.413549 20334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:51:19.413667 20334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:51:19.414711 20137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } }
I20260927 18:51:19.415788 20137 leader_election.cc:288] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [CANDIDATE]: Term 1 election: Requested vote from peers 229da1707d034573949dc89cba9f62a9 (127.29.159.125:44369), 50f2d7d23ef7444ca8271d72e311199a (127.29.159.124:37331)
I20260927 18:51:19.416746 20314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:19.414317 20095 master_runner.cc:186] Error getting master registration for 127.29.159.125:44369: OK, instance_id { permanent_uuid: "229da1707d034573949dc89cba9f62a9" instance_seqno: 1790535079132380 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:51:19.417243 20095 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:51:19.417341 20238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } }
I20260927 18:51:19.417586 20238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229da1707d034573949dc89cba9f62a9, State: Initialized, Role: FOLLOWER
I20260927 18:51:19.417943 20314 log.cc:824] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:19.417961 20238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } }
I20260927 18:51:19.419559 20238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:19.420496 20314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a: No bootstrap required, opened a new log
I20260927 18:51:19.422981 20212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5460791c78b34e51b9697ec981919160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "229da1707d034573949dc89cba9f62a9"
I20260927 18:51:19.424695 20212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:19.426074 20212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5460791c78b34e51b9697ec981919160 in term 1.
W20260927 18:51:19.426288 20355 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:51:19.426369 20355 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:51:19.426743 20114 leader_election.cc:302] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 229da1707d034573949dc89cba9f62a9, 5460791c78b34e51b9697ec981919160; no voters:
I20260927 18:51:19.429337 20341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } } }
I20260927 18:51:19.429500 20341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:19.431298 20170 master_runner.cc:186] Error getting master registration for 127.29.159.124:37331: OK, instance_id { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" instance_seqno: 1790535079355498 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:51:19.434893 20323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:19.435190 20323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [term 1 LEADER]: Becoming Leader. State: Replica: 5460791c78b34e51b9697ec981919160, State: Running, Role: LEADER
I20260927 18:51:19.435617 20323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } }
I20260927 18:51:19.438146 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.418255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.29.159.125:44369)
0927 18:51:19.418408 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.418413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.420284 (+ 1871us) server_negotiation.cc:407] Connection header received
0927 18:51:19.420530 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.420535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.420598 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.420665 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.423668 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.424498 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.427534 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.427974 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.428066 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.432781 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.432785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.432788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.432817 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.435861 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.436466 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.436472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.436477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.436576 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.436940 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.436944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.436950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.437231 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.437365 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.437787 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.437936 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:51:19.439605 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.419662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.29.159.124:37331)
0927 18:51:19.419782 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.419787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.426768 (+ 6981us) server_negotiation.cc:407] Connection header received
0927 18:51:19.427096 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.427100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.427153 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.427207 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.428300 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.429125 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.430427 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.430838 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.430950 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.433721 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.433728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.433733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.433766 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.436096 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.437895 (+ 1799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.437901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.437904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.438044 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.438432 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.438437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.438440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.438690 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.438838 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.439237 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.439364 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:51:19.440001 20323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5460791c78b34e51b9697ec981919160. Latest consensus state: current_term: 1 leader_uuid: "5460791c78b34e51b9697ec981919160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } } }
I20260927 18:51:19.440277 20323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:19.441033 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:19.418913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.29.159.126:46043)
0927 18:51:19.419028 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.419032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.422030 (+ 2998us) server_negotiation.cc:407] Connection header received
0927 18:51:19.422250 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.422255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.422312 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.422371 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.425777 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.426580 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.430992 (+ 4412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:19.431365 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:19.431496 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:19.434204 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:19.434209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:19.434214 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:19.434249 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:19.437008 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.439472 (+ 2464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.439477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.439480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.439579 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:19.439947 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:19.439951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:19.439952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:19.440211 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:19.440320 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:19.440707 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 18:51:19.440854 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:51:19.443519 20314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } }
I20260927 18:51:19.443849 20314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f2d7d23ef7444ca8271d72e311199a, State: Initialized, Role: FOLLOWER
I20260927 18:51:19.444257 20314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } }
I20260927 18:51:19.445744 20314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:19.446091 20288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5460791c78b34e51b9697ec981919160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "50f2d7d23ef7444ca8271d72e311199a"
I20260927 18:51:19.446355 20288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:19.447515 20170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:51:19.447973 20288 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5460791c78b34e51b9697ec981919160 in term 1.
I20260927 18:51:19.448638 20360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } } }
I20260927 18:51:19.448779 20360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:19.450726 20359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:19.456081 20359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 18:51:19.461248 20371 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:51:19.461406 20371 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:51:19.466262 20288 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [term 1 FOLLOWER]: Refusing update from remote peer 5460791c78b34e51b9697ec981919160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:51:19.467067 20357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:51:19.475100 20360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5460791c78b34e51b9697ec981919160. Latest consensus state: current_term: 1 leader_uuid: "5460791c78b34e51b9697ec981919160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } } }
I20260927 18:51:19.475229 20360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:19.476387 20212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [term 1 FOLLOWER]: Refusing update from remote peer 5460791c78b34e51b9697ec981919160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:51:19.476998 20357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:51:19.481050 20341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5460791c78b34e51b9697ec981919160. Latest consensus state: current_term: 1 leader_uuid: "5460791c78b34e51b9697ec981919160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } } }
I20260927 18:51:19.481187 20341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:19.486620 20357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5460791c78b34e51b9697ec981919160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } } }
I20260927 18:51:19.486773 20357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:19.488276 20360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5460791c78b34e51b9697ec981919160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } } }
I20260927 18:51:19.488435 20360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:19.488811 20373 mvcc.cc:205] Tried to move back new op lower bound from 7334031685485318144 to 7334031685374406656. Current Snapshot: MvccSnapshot[applied={T|T < 7334031685485318144}]
I20260927 18:51:19.491655 20374 mvcc.cc:205] Tried to move back new op lower bound from 7334031685485318144 to 7334031685374406656. Current Snapshot: MvccSnapshot[applied={T|T < 7334031685485318144}]
I20260927 18:51:19.494582 20359 catalog_manager.cc:1347] Generated new cluster ID: a4197c38e16442afa2c119fbfba3d563
I20260927 18:51:19.494661 20359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:19.495739 20323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5460791c78b34e51b9697ec981919160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } } }
I20260927 18:51:19.495862 20323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:19.497641 20376 mvcc.cc:205] Tried to move back new op lower bound from 7334031685485318144 to 7334031685374406656. Current Snapshot: MvccSnapshot[applied={T|T < 7334031685485318144}]
I20260927 18:51:19.498647 20341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5460791c78b34e51b9697ec981919160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5460791c78b34e51b9697ec981919160" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46043 } } peers { permanent_uuid: "229da1707d034573949dc89cba9f62a9" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44369 } } peers { permanent_uuid: "50f2d7d23ef7444ca8271d72e311199a" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 37331 } } }
I20260927 18:51:19.498773 20341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:19.522055 20359 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:19.523634 20359 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:19.543823 20359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5460791c78b34e51b9697ec981919160: Generated new TSK 0
I20260927 18:51:19.544816 20359 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:51:19.583770 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.579991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.29.159.125:44369)
0927 18:51:19.580156 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.580161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.582472 (+ 2311us) server_negotiation.cc:407] Connection header received
0927 18:51:19.582720 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.582724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.582780 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.582842 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.583613 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45704: BlockingRecv error: recv got EOF from 127.0.0.1:45704 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 18:51:19.585707 20319 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.581810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.29.159.124:37331)
0927 18:51:19.581947 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.581952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.584623 (+ 2671us) server_negotiation.cc:407] Connection header received
0927 18:51:19.584815 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.584819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.584877 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.584940 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.585566 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53964: BlockingRecv error: recv got EOF from 127.0.0.1:53964 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 18:51:19.587209 20166 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.580832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.29.159.126:46043)
0927 18:51:19.580968 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.580972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.584581 (+ 3609us) server_negotiation.cc:407] Connection header received
0927 18:51:19.586423 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.586427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.586495 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.586592 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.587067 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60234: BlockingRecv error: recv got EOF from 127.0.0.1:60234 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 18:51:19.592245 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.590327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.29.159.125:44369)
0927 18:51:19.590463 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.590468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.591117 (+ 649us) server_negotiation.cc:407] Connection header received
0927 18:51:19.591377 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.591381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.591497 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.591558 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.592134 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45710: BlockingRecv error: recv got EOF from 127.0.0.1:45710 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 18:51:19.598083 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.596251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.29.159.125:44369)
0927 18:51:19.596383 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.596388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.597006 (+ 618us) server_negotiation.cc:407] Connection header received
0927 18:51:19.597241 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.597246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.597306 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.597378 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.597971 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45712: BlockingRecv error: recv got EOF from 127.0.0.1:45712 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 18:51:19.604171 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.602396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.29.159.125:44369)
0927 18:51:19.602516 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.602521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.603111 (+ 590us) server_negotiation.cc:407] Connection header received
0927 18:51:19.603318 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.603322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.603395 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.603485 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.604047 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45722: BlockingRecv error: recv got EOF from 127.0.0.1:45722 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 18:51:19.612541 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.610572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.29.159.125:44369)
0927 18:51:19.610717 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.610722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.611359 (+ 637us) server_negotiation.cc:407] Connection header received
0927 18:51:19.611620 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.611624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.611686 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.611747 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.612401 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45726: BlockingRecv error: recv got EOF from 127.0.0.1:45726 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 18:51:19.628937 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.623302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.29.159.125:44369)
0927 18:51:19.623484 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.623490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.627802 (+ 4312us) server_negotiation.cc:407] Connection header received
0927 18:51:19.628041 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.628046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.628103 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.628162 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.628797 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45730: BlockingRecv error: recv got EOF from 127.0.0.1:45730 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 18:51:19.649104 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.647755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.29.159.125:44369)
0927 18:51:19.647893 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.647899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.648174 (+ 275us) server_negotiation.cc:407] Connection header received
0927 18:51:19.648344 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.648348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.648413 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.648504 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.648982 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45736: BlockingRecv error: recv got EOF from 127.0.0.1:45736 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 18:51:19.684341 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.683033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.29.159.125:44369)
0927 18:51:19.683163 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.683167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.683410 (+ 243us) server_negotiation.cc:407] Connection header received
0927 18:51:19.683605 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.683613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.683662 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.683747 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.684211 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45740: BlockingRecv error: recv got EOF from 127.0.0.1:45740 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 18:51:19.751588 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.750463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.29.159.125:44369)
0927 18:51:19.750634 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.750637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.750743 (+ 106us) server_negotiation.cc:407] Connection header received
0927 18:51:19.750868 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.750871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.750918 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.751003 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.751393 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45754: BlockingRecv error: recv got EOF from 127.0.0.1:45754 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 18:51:19.883129 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:19.881783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.29.159.125:44369)
0927 18:51:19.882044 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:19.882049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:19.882173 (+ 124us) server_negotiation.cc:407] Connection header received
0927 18:51:19.882365 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:19.882369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:19.882435 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:19.882529 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:19.883000 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45756: BlockingRecv error: recv got EOF from 127.0.0.1:45756 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
W20260927 18:51:20.142711 20244 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:20.141536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.29.159.125:44369)
0927 18:51:20.141681 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:20.141686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:20.141774 (+ 88us) server_negotiation.cc:407] Connection header received
0927 18:51:20.141884 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:20.141888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:20.141943 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:20.142044 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:20.142572 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45762: BlockingRecv error: recv got EOF from 127.0.0.1:45762 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:51:20.427711 20355 catalog_manager.cc:1259] Loaded cluster ID: a4197c38e16442afa2c119fbfba3d563
I20260927 18:51:20.427799 20355 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9: loading cluster ID for follower catalog manager: success
I20260927 18:51:20.429507 20355 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9: acquiring CA information for follower catalog manager: success
I20260927 18:51:20.430713 20355 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 229da1707d034573949dc89cba9f62a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:51:20.462796 20371 catalog_manager.cc:1259] Loaded cluster ID: a4197c38e16442afa2c119fbfba3d563
I20260927 18:51:20.462901 20371 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a: loading cluster ID for follower catalog manager: success
I20260927 18:51:20.464771 20371 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a: acquiring CA information for follower catalog manager: success
I20260927 18:51:20.466017 20371 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50f2d7d23ef7444ca8271d72e311199a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:51:20.661085 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:20.656865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.29.159.125:44369)
0927 18:51:20.657152 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:20.657157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:20.657173 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:51:20.657511 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:20.657515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:20.657650 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:20.657767 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:51:20.658055 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:20.658602 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:20.659545 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:20.659772 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:20.659893 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:20.660212 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:51:20.660665 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:51:20.660821 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:20.660865 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 18:51:20.660923 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260927 18:51:20.668540 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:20.664339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.29.159.124:37331)
0927 18:51:20.664616 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:20.664620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:20.664874 (+ 254us) server_negotiation.cc:407] Connection header received
0927 18:51:20.664984 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:20.664986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:20.665102 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:20.665197 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:51:20.665394 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:20.665919 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:20.666734 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:20.666997 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:20.667083 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:20.667424 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:51:20.668100 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:51:20.668246 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:20.668322 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 18:51:20.668383 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":76,"threads_started":1}
I20260927 18:51:20.674993 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:20.671682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.29.159.124:37331)
0927 18:51:20.671815 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:20.671819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:20.671833 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:51:20.671962 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:20.671966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:20.672040 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:20.672114 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:51:20.672439 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:20.673021 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:20.673939 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:20.674170 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:20.674275 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:20.674509 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:51:20.674581 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:51:20.674659 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:20.674773 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 18:51:20.674826 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:51:20.676152 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:20.671372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.29.159.125:44369)
0927 18:51:20.671527 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:20.671531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:20.672153 (+ 622us) server_negotiation.cc:407] Connection header received
0927 18:51:20.672243 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:20.672246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:20.672307 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:20.672376 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:51:20.672748 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:20.673532 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:20.674900 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:20.675233 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:20.675308 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:20.675722 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:51:20.675801 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:51:20.675874 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:20.675990 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 18:51:20.676059 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:51:20.676908 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:20.671618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.29.159.126:46043)
0927 18:51:20.671875 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:20.671879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:20.671893 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:51:20.672072 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:20.672075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:20.672242 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:20.672375 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:51:20.672604 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:20.673450 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:20.674897 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:20.675525 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:20.675912 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:20.676068 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:51:20.676473 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:51:20.676556 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:20.676681 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 18:51:20.676732 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
I20260927 18:51:20.687278 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 20095
I20260927 18:51:20.694024 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 20170
I20260927 18:51:20.699965 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 20245
2026-09-27T18:51:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3131 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:51:20 dist-test-slave-8tg2 krb5kdc[20463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:51:20 dist-test-slave-8tg2 krb5kdc[20463](info): set up 2 sockets
Sep 27 18:51:20 dist-test-slave-8tg2 krb5kdc[20463](info): commencing operation
krb5kdc: starting...
W20260927 18:51:21.747941 30333 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/test-user.keytab.
Sep 27 18:51:21 dist-test-slave-8tg2 krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.159.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
Entry for principal kudu/127.29.159.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab.
I20260927 18:51:21.874841 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:46723
--webserver_interface=127.29.159.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:46723,127.29.159.125:44131,127.29.159.124:33909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:51:21.989179 20476 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:51:21.989543 20476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:21.989616 20476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:21.989697 20476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:51:21.993963 20476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:21.994068 20476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:21.994103 20476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:21.994148 20476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:21.994180 20476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:21.999742 20476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:46723,127.29.159.125:44131,127.29.159.124:33909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.126.keytab
--principal=kudu/127.29.159.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.126:46723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.159.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:22.000910 20476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:22.001991 20476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:22.010654 20484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:22.010654 20481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:22.010684 20482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:22.011247 20476 server_base.cc:1034] running on GCE node
I20260927 18:51:22.011883 20476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:22.013054 20476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:22.014256 20476 hybrid_clock.cc:648] HybridClock initialized: now 1790535082014240 us; error 29 us; skew 500 ppm
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:22.017138 20476 init.cc:377] Logged in from keytab as kudu/127.29.159.126@KRBTEST.COM (short username kudu)
I20260927 18:51:22.018113 20476 webserver.cc:460] Webserver started at http://127.29.159.126:34229/ using document root <none> and password file <none>
I20260927 18:51:22.018702 20476 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:22.018791 20476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:22.019028 20476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:22.021219 20476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/data/instance:
uuid: "a72c292436404388be4ea4803d19f096"
format_stamp: "Formatted at 2026-09-27 18:51:22 on dist-test-slave-8tg2"
server_key: "f6fb34a98c2769780ff64d0656cc17ed"
server_key_iv: "3c12826bc96f6bdfc8b61c4e7c42100f"
server_key_version: "encryptionkey@0"
I20260927 18:51:22.021799 20476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/wal/instance:
uuid: "a72c292436404388be4ea4803d19f096"
format_stamp: "Formatted at 2026-09-27 18:51:22 on dist-test-slave-8tg2"
server_key: "f6fb34a98c2769780ff64d0656cc17ed"
server_key_iv: "3c12826bc96f6bdfc8b61c4e7c42100f"
server_key_version: "encryptionkey@0"
I20260927 18:51:22.025655 20476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 18:51:22.028203 20491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:22.029485 20476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:51:22.029620 20476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/wal
uuid: "a72c292436404388be4ea4803d19f096"
format_stamp: "Formatted at 2026-09-27 18:51:22 on dist-test-slave-8tg2"
server_key: "f6fb34a98c2769780ff64d0656cc17ed"
server_key_iv: "3c12826bc96f6bdfc8b61c4e7c42100f"
server_key_version: "encryptionkey@0"
I20260927 18:51:22.029733 20476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:22.058000 20476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:22.061189 20476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:22.061408 20476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:22.068784 20476 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.126:46723
I20260927 18:51:22.069093 20543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.126:46723 every 8 connection(s)
I20260927 18:51:22.069847 20476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/data/info.pb
I20260927 18:51:22.071862 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 20476
I20260927 18:51:22.071969 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-0/wal/instance
I20260927 18:51:22.072265 30333 external_mini_cluster.cc:1428] Setting key dcd11e83a60d435225dc672c7ce63dc7
I20260927 18:51:22.072858 20544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:22.079164 20544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.159.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.125@KRBTEST.COM" created.
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:51:22.101334 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.082995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46723 (local address 127.0.0.1:36510)
0927 18:51:22.083881 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.083933 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:22.085299 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:22.086236 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:22.086246 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:22.086774 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:22.087236 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.087263 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.089622 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.089627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.090963 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.090976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.092031 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.092034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.092116 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.092797 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:22.092826 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:22.094975 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:22.097385 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.097396 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.097409 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.097743 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.098150 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.098152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.098155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.098273 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.099123 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:22.099131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:22.099330 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 18:51:22.099633 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 18:51:22.100164 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":107,"spinlock_wait_cycles":58624,"thread_start_us":168,"threads_started":1}
I20260927 18:51:22.101341 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.082995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.29.159.126:46723)
0927 18:51:22.083551 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.083574 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.084048 (+ 474us) server_negotiation.cc:407] Connection header received
0927 18:51:22.085719 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.085727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.085989 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.086215 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.087475 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.089475 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.091099 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.091847 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.092031 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.095164 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.095174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.095192 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.095218 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.097294 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.097899 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.097909 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.097911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.098032 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.098395 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.098399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.098400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.098924 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.099064 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.099751 (+ 687us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.100068 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":331,"thread_start_us":149,"threads_started":1}
I20260927 18:51:22.104537 20544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab.
Entry for principal kudu/127.29.159.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab.
I20260927 18:51:22.105473 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.105113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44131 (local address 127.0.0.1:46614)
0927 18:51:22.105303 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.105392 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 18:51:22.105695 20493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44131: connect: Connection refused (error 111)
I20260927 18:51:22.106693 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.125:44131
--webserver_interface=127.29.159.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab
--principal=kudu/127.29.159.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:46723,127.29.159.125:44131,127.29.159.124:33909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:51:22.108623 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.106383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44131 (local address 127.0.0.1:46618)
0927 18:51:22.108492 (+ 2109us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.108555 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2016}
W20260927 18:51:22.108836 20544 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.125:44131: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44131: connect: Connection refused (error 111)
I20260927 18:51:22.158136 20544 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } attempt: 1
I20260927 18:51:22.159049 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.158736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44131 (local address 127.0.0.1:46628)
0927 18:51:22.158884 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.158956 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 18:51:22.159861 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.159653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44131 (local address 127.0.0.1:46632)
0927 18:51:22.159750 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.159798 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 18:51:22.160028 20544 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.125:44131: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44131: connect: Connection refused (error 111)
W20260927 18:51:22.227738 20551 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:51:22.228036 20551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:22.228091 20551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:22.228166 20551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 18:51:22.228210 20544 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } attempt: 2
I20260927 18:51:22.229020 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.228730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44131 (local address 127.0.0.1:46638)
0927 18:51:22.228871 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.228938 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 18:51:22.229727 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.229534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44131 (local address 127.0.0.1:46650)
0927 18:51:22.229628 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.229674 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 18:51:22.229864 20544 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.125:44131: Network error: Client connection negotiation failed: client connection to 127.29.159.125:44131: connect: Connection refused (error 111)
W20260927 18:51:22.232684 20551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:22.232736 20551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:22.232774 20551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:22.232817 20551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:22.232843 20551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:22.238397 20551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:46723,127.29.159.125:44131,127.29.159.124:33909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.125.keytab
--principal=kudu/127.29.159.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.125:44131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.159.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:22.239530 20551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:22.240568 20551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:22.249158 20557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:22.249186 20551 server_base.cc:1034] running on GCE node
W20260927 18:51:22.249140 20556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:22.249122 20559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:22.249996 20551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:22.251045 20551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:22.252238 20551 hybrid_clock.cc:648] HybridClock initialized: now 1790535082252217 us; error 33 us; skew 500 ppm
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:22.255410 20551 init.cc:377] Logged in from keytab as kudu/127.29.159.125@KRBTEST.COM (short username kudu)
I20260927 18:51:22.256474 20551 webserver.cc:460] Webserver started at http://127.29.159.125:36581/ using document root <none> and password file <none>
I20260927 18:51:22.257067 20551 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:22.257133 20551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:22.257364 20551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:22.259619 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/data/instance:
uuid: "db915133f79a4770a095673dd1f7bbdc"
format_stamp: "Formatted at 2026-09-27 18:51:22 on dist-test-slave-8tg2"
server_key: "705aac044d66d788edb37fb79b96c390"
server_key_iv: "996331cc696e380f59a0668eb71f1132"
server_key_version: "encryptionkey@0"
I20260927 18:51:22.260196 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/wal/instance:
uuid: "db915133f79a4770a095673dd1f7bbdc"
format_stamp: "Formatted at 2026-09-27 18:51:22 on dist-test-slave-8tg2"
server_key: "705aac044d66d788edb37fb79b96c390"
server_key_iv: "996331cc696e380f59a0668eb71f1132"
server_key_version: "encryptionkey@0"
I20260927 18:51:22.264405 20551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:51:22.266987 20566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:22.268328 20551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 18:51:22.268442 20551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/wal
uuid: "db915133f79a4770a095673dd1f7bbdc"
format_stamp: "Formatted at 2026-09-27 18:51:22 on dist-test-slave-8tg2"
server_key: "705aac044d66d788edb37fb79b96c390"
server_key_iv: "996331cc696e380f59a0668eb71f1132"
server_key_version: "encryptionkey@0"
I20260927 18:51:22.268610 20551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:22.289588 20551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:22.292943 20551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:22.293159 20551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:22.299995 20551 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.125:44131
I20260927 18:51:22.300271 20618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.125:44131 every 8 connection(s)
I20260927 18:51:22.301038 20551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/data/info.pb
I20260927 18:51:22.303850 20619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:22.304646 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 20551
I20260927 18:51:22.304786 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-1/wal/instance
I20260927 18:51:22.305097 30333 external_mini_cluster.cc:1428] Setting key 5a70862e674cfda2c799559db1bce9ba
I20260927 18:51:22.308507 20619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.159.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.159.124@KRBTEST.COM" created.
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:51:22.321025 20544 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } attempt: 3
I20260927 18:51:22.323415 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.310500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.29.159.126:46723)
0927 18:51:22.310651 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.310655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.311823 (+ 1168us) server_negotiation.cc:407] Connection header received
0927 18:51:22.312762 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.312765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.312828 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.312931 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.314079 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.314605 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.315777 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.316059 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.316168 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.318846 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.318853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.318859 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.318894 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.320661 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.321168 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.321174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.321177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.321284 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.322015 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.322019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.322020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.322232 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.322344 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.323063 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.323224 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:51:22.324599 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.311082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46723 (local address 127.0.0.1:36522)
0927 18:51:22.311668 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.311711 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:22.312559 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:22.313091 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:22.313098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:22.313476 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:22.313940 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.313952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.314728 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.314736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.315663 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.315673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.316204 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.316209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.316379 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.317025 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:22.317042 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:22.318663 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:22.320782 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.320786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.320797 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.321055 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.321376 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.321382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.321385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.321914 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.322369 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:22.322378 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:22.322727 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 18:51:22.322957 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 18:51:22.323458 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":134,"threads_started":1}
I20260927 18:51:22.326099 20619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.159.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab.
Entry for principal kudu/127.29.159.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab.
I20260927 18:51:22.332544 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.321503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44131 (local address 127.0.0.1:46660)
0927 18:51:22.321666 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.321683 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:22.321763 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:22.322375 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:22.322377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:22.322674 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:22.323301 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.323311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.325016 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.325023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.325742 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.325751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.326448 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.326453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.326590 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.327085 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:22.327124 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:22.328537 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:22.330833 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.330836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.330838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.331148 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.331472 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.331479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.331481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.331573 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.332146 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:22.332148 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:22.332252 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 18:51:22.332333 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:51:22.332433 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 18:51:22.332847 30333 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskclkeqe/build/debug/bin/kudu
/tmp/dist-test-taskclkeqe/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.124:33909
--webserver_interface=127.29.159.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab
--principal=kudu/127.29.159.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.159.84:39009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.159.126:46723,127.29.159.125:44131,127.29.159.124:33909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:51:22.335060 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.321594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.29.159.125:44131)
0927 18:51:22.321937 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.321946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.321982 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:51:22.322025 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.322031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.322236 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.323890 (+ 1654us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.323996 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.324891 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.325892 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.326316 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.326811 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.328669 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.328677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.328687 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.328706 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.330715 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.331257 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.331261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.331263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.331360 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.331683 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.331690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.331694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.332027 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.334611 (+ 2584us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.334728 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.334866 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":156,"threads_started":1}
I20260927 18:51:22.336905 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.326659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44131 (local address 127.0.0.1:46672)
0927 18:51:22.326765 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.326781 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:22.326854 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:22.327364 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:22.327367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:22.327561 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:22.327634 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.327639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.328540 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.328544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.329333 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.329343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.329996 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.330004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.330078 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.330468 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:22.330486 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:22.332154 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:22.334916 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.334919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.334921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.335360 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.335839 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.335841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.335842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.335948 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.336389 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:22.336392 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:22.336525 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 18:51:22.336629 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:51:22.336746 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 18:51:22.337141 20544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:51:22.337224 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.326847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.29.159.125:44131)
0927 18:51:22.327183 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.327190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.327201 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:51:22.327226 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.327228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.327258 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.327351 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.327776 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.328435 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.329492 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.329877 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.329997 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.332303 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.332306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.332310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.332332 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.334808 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.335616 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.335619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.335621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.335720 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.336053 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.336058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.336062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.336291 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.336394 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.336722 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.336854 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260927 18:51:22.337800 20619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:51:22.337904 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.337602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54522)
0927 18:51:22.337785 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.337838 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 18:51:22.338452 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.338160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54532)
0927 18:51:22.338277 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.338378 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 18:51:22.338593 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.338349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54538)
0927 18:51:22.338484 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.338538 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 18:51:22.338758 20544 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:33909: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
W20260927 18:51:22.338816 20568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
I20260927 18:51:22.339658 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.339373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54550)
0927 18:51:22.339523 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.339585 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 18:51:22.339824 20619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:33909: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
I20260927 18:51:22.369910 20544 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } attempt: 1
I20260927 18:51:22.371466 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.370499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54566)
0927 18:51:22.371251 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.371331 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587}
I20260927 18:51:22.372260 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.372057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54576)
0927 18:51:22.372150 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.372198 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 18:51:22.372404 20544 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:33909: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
I20260927 18:51:22.389075 20619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } attempt: 1
I20260927 18:51:22.389938 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.389642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54584)
0927 18:51:22.389788 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.389860 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 18:51:22.390686 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.390414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54596)
0927 18:51:22.390542 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.390608 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 18:51:22.390822 20619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:33909: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
I20260927 18:51:22.447552 20544 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } attempt: 2
I20260927 18:51:22.448414 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.448125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54604)
0927 18:51:22.448272 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.448334 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 18:51:22.449131 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.448910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54616)
0927 18:51:22.449028 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.449079 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 18:51:22.449252 20544 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:33909: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
W20260927 18:51:22.457567 20626 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:51:22.457935 20626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:51:22.458001 20626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:51:22.458097 20626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 18:51:22.458969 20619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } attempt: 2
I20260927 18:51:22.459784 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.459509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54622)
0927 18:51:22.459643 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.459710 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 18:51:22.460516 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.460312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54636)
0927 18:51:22.460411 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.460456 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 18:51:22.460671 20619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.159.124:33909: Network error: Client connection negotiation failed: client connection to 127.29.159.124:33909: connect: Connection refused (error 111)
W20260927 18:51:22.463129 20626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:51:22.463219 20626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:51:22.463289 20626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:51:22.463361 20626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:51:22.463402 20626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:51:22.469558 20626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.159.84:39009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.159.126:46723,127.29.159.125:44131,127.29.159.124:33909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/krb5kdc/kudu_127.29.159.124.keytab
--principal=kudu/127.29.159.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.159.124:33909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.159.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 17:58:57 UTC on ccycloud.quasar-mffepp.root.comops.site
build id 4676274
I20260927 18:51:22.470829 20626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:51:22.472105 20626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:51:22.481189 20632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:51:22.481205 20635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:22.481531 20626 server_base.cc:1034] running on GCE node
W20260927 18:51:22.481577 20633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:51:22.482091 20626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:51:22.483227 20626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:51:22.484411 20626 hybrid_clock.cc:648] HybridClock initialized: now 1790535082484391 us; error 43 us; skew 500 ppm
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:51:22.487228 20626 init.cc:377] Logged in from keytab as kudu/127.29.159.124@KRBTEST.COM (short username kudu)
I20260927 18:51:22.488184 20626 webserver.cc:460] Webserver started at http://127.29.159.124:42935/ using document root <none> and password file <none>
I20260927 18:51:22.488771 20626 fs_manager.cc:359] Metadata directory not provided
I20260927 18:51:22.488859 20626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:51:22.489099 20626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:51:22.491308 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/data/instance:
uuid: "24c35297ea70497da8638174d3955925"
format_stamp: "Formatted at 2026-09-27 18:51:22 on dist-test-slave-8tg2"
server_key: "58c3ea120bbde9178d98f149804c60c8"
server_key_iv: "5c1067a65f42771e9a2f85db6a2fc86c"
server_key_version: "encryptionkey@0"
I20260927 18:51:22.491930 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/wal/instance:
uuid: "24c35297ea70497da8638174d3955925"
format_stamp: "Formatted at 2026-09-27 18:51:22 on dist-test-slave-8tg2"
server_key: "58c3ea120bbde9178d98f149804c60c8"
server_key_iv: "5c1067a65f42771e9a2f85db6a2fc86c"
server_key_version: "encryptionkey@0"
I20260927 18:51:22.495621 20626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:51:22.498051 20642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:51:22.499346 20626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:51:22.499517 20626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/data,/tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/wal
uuid: "24c35297ea70497da8638174d3955925"
format_stamp: "Formatted at 2026-09-27 18:51:22 on dist-test-slave-8tg2"
server_key: "58c3ea120bbde9178d98f149804c60c8"
server_key_iv: "5c1067a65f42771e9a2f85db6a2fc86c"
server_key_version: "encryptionkey@0"
I20260927 18:51:22.499629 20626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:51:22.517493 20626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:51:22.520813 20626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:51:22.521021 20626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:51:22.527843 20626 rpc_server.cc:307] RPC server started. Bound to: 127.29.159.124:33909
I20260927 18:51:22.528172 20694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.159.124:33909 every 8 connection(s)
I20260927 18:51:22.528815 20626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/data/info.pb
I20260927 18:51:22.530764 30333 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskclkeqe/build/debug/bin/kudu as pid 20626
I20260927 18:51:22.530905 30333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskclkeqe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534801142164-30333-0/minicluster-data/master-2/wal/instance
I20260927 18:51:22.531215 30333 external_mini_cluster.cc:1428] Setting key 72e9c0382197c33da7b2db63aa664ae2
I20260927 18:51:22.531896 20695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:51:22.537153 20695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
I20260927 18:51:22.546832 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.532611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.29.159.124:33909)
0927 18:51:22.533027 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.533042 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.533077 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:51:22.533872 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.533891 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.534258 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.534606 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.535565 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.537048 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.538184 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.538685 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.538842 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.541664 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.541669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.541678 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.541704 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.543635 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.544039 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.544045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.544052 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.544194 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.544480 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.544484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.544486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.544968 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.545109 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.545488 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.545796 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":134,"threads_started":1}
I20260927 18:51:22.548238 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.539092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.29.159.126:46723)
0927 18:51:22.539248 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.539253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.539640 (+ 387us) server_negotiation.cc:407] Connection header received
0927 18:51:22.539764 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.539767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.539840 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.539936 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.540568 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.541052 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.542072 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.542355 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.542493 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.544759 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.544762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.544764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.544786 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.546356 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.546795 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.546798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.546799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.546891 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.547245 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.547253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.547255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.547535 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.547639 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.547973 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.548081 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:51:22.548256 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.539054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.126:46723 (local address 127.0.0.1:36534)
0927 18:51:22.539483 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.539553 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:22.539669 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:22.539985 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:22.539989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:22.540318 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:22.540439 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.540445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.541181 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.541185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.541964 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.541974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.542497 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.542501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.542642 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.543064 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:22.543075 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:22.544609 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:22.546467 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.546475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.546479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.546702 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.547015 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.547021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.547024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.547134 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.547640 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:22.547647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:22.547759 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 18:51:22.547903 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 18:51:22.548081 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":158,"threads_started":1}
I20260927 18:51:22.548595 20544 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } attempt: 3
I20260927 18:51:22.549711 20695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:51:22.551808 20619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } attempt: 3
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.126@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.126@KRBTEST.COM
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.125@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
I20260927 18:51:22.560020 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.549052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54648)
0927 18:51:22.549166 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.549181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:22.549257 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:22.549485 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:22.549488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:22.549660 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:22.549756 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.549763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.550714 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.550720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.551521 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.551530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.552339 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.552343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.552452 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.552983 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:22.553000 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:22.555115 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:22.558083 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.558087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.558090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.558549 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.558878 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.558884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.558888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.558979 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.559466 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:22.559471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:22.559597 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 18:51:22.559682 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 18:51:22.559839 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1,"spinlock_wait_cycles":2944}
I20260927 18:51:22.560592 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.550318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.29.159.125:44131)
0927 18:51:22.550463 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.550468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.550634 (+ 166us) server_negotiation.cc:407] Connection header received
0927 18:51:22.550791 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.550796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.550859 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.550953 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.551380 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.552073 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.553123 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.553472 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.553541 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.556280 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.556283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.556286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.556314 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.558587 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.559110 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.559113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.559114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.559184 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.559544 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.559548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.559550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.559747 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.559816 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.560315 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.560440 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:51:22.560770 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.549109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.29.159.124:33909)
0927 18:51:22.549222 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.549227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.549241 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:51:22.549333 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.549337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.549383 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.549460 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.549902 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.550571 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.551625 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.552211 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.555520 (+ 3309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.555611 (+ 91us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.555613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.555615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.555627 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.557947 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.558680 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.558685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.558686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.558770 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.559099 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.559102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.559104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.559317 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.560320 (+ 1003us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.560385 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.560523 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 18:51:22.560956 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.549662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.29.159.126:46723)
0927 18:51:22.549742 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.549746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.549857 (+ 111us) server_negotiation.cc:407] Connection header received
0927 18:51:22.550306 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.550309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.550363 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.550445 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.551164 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.551788 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.552939 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.553244 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.553308 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.556463 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.556468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.556471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.556495 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.558246 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.559059 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.559062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.559064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.559148 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.559549 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.559554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.559557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.559819 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.560054 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.560645 (+ 591us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.560767 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25,"spinlock_wait_cycles":1408}
I20260927 18:51:22.561229 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.550210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.125:44131 (local address 127.0.0.1:46686)
0927 18:51:22.550572 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.550582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:22.550664 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:22.550980 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:22.550984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:22.551146 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:22.551249 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.551255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.552208 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.552211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.553011 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.553020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.553700 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.553706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.553813 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.554225 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:22.554241 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:22.556154 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:22.558746 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.558749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.558751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.558999 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.559291 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.559294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.559296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.559385 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.559888 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:22.559891 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:22.559984 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 18:51:22.560969 (+ 985us) client_negotiation.cc:240] Negotiation successful
0927 18:51:22.561097 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"spinlock_wait_cycles":6144}
I20260927 18:51:22.562121 20695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:51:22.564401 20544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096: Bootstrap starting.
I20260927 18:51:22.567387 20544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096: Neither blocks nor log segments found. Creating new log.
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535082, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.159.124@KRBTEST.COM for kudu/127.29.159.124@KRBTEST.COM
Sep 27 18:51:22 dist-test-slave-8tg2 krb5kdc[20463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790535081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.159.125@KRBTEST.COM
I20260927 18:51:22.568255 20544 log.cc:824] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:22.569882 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.552159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54664)
0927 18:51:22.552381 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.552394 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:22.552459 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:22.553201 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:22.553204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:22.553371 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:22.553463 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.553473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.555153 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.555157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.556039 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.556050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.556647 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.556653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.556757 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.557192 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:22.557209 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:22.558941 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:22.561128 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.561131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.561133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.561383 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.568738 (+ 7355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.568741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.568744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.568858 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.569361 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:22.569365 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:22.569463 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 18:51:22.569564 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:51:22.569712 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260927 18:51:22.569942 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.552592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.29.159.124:33909)
0927 18:51:22.552960 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.552969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.552982 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:51:22.553014 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.553016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.553059 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.553586 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.553693 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.555019 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.556192 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.556544 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.556648 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.559078 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.559081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.559083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.559109 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.560938 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.563605 (+ 2667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.563609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.563611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.563699 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.568992 (+ 5293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.568997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.568999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.569254 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.569371 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.569632 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.569758 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":112,"threads_started":1}
I20260927 18:51:22.572088 20544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096: No bootstrap required, opened a new log
I20260927 18:51:22.572468 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.562461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.29.159.125:44131)
0927 18:51:22.562571 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.562575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.562592 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:51:22.562722 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.562725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.562768 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.562852 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.563723 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.564426 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.565466 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.565795 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.565889 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.568677 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.568680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.568682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.568704 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.570497 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.570980 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.570983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.570985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.571064 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.571365 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.571381 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.571383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.571605 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.571790 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.572195 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.572303 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:51:22.572494 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.562558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.159.124:33909 (local address 127.0.0.1:54670)
0927 18:51:22.562871 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 18:51:22.562884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:51:22.562954 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:51:22.563956 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:51:22.563958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:51:22.564077 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:51:22.564133 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.564138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.564878 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.564884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.565372 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:51:22.565378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.565996 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.565999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:51:22.566048 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.566600 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:51:22.566611 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:51:22.567885 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:51:22.570944 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.570947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.570949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.571176 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.571472 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:51:22.571475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:51:22.571476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:51:22.571549 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:51:22.571934 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:51:22.571936 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:51:22.572012 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 18:51:22.572199 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 18:51:22.572324 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260927 18:51:22.573238 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.563588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.29.159.124:33909)
0927 18:51:22.563690 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.563693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.563704 (+ 11us) server_negotiation.cc:407] Connection header received
0927 18:51:22.563733 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.563738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.563782 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.563845 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.564234 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.564783 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.565504 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.565803 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.565885 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.568031 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.568035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.568037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.568060 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.570828 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.571279 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.571284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.571289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.571366 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.571638 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.571641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.571642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.571814 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.571893 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.572961 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.573061 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:51:22.573578 20619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc: Bootstrap starting.
I20260927 18:51:22.575696 20544 raft_consensus.cc:348] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } }
I20260927 18:51:22.575906 20544 raft_consensus.cc:729] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72c292436404388be4ea4803d19f096, State: Initialized, Role: FOLLOWER
I20260927 18:51:22.576164 20695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925: Bootstrap starting.
I20260927 18:51:22.576277 20619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:22.576202 20544 consensus_queue.cc:260] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } }
I20260927 18:51:22.576676 20703 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } } }
I20260927 18:51:22.576830 20703 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:22.577164 20619 log.cc:824] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:22.577502 20544 sys_catalog.cc:564] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:22.579180 20695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925: Neither blocks nor log segments found. Creating new log.
I20260927 18:51:22.580088 20695 log.cc:824] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925: Log is configured to *not* fsync() on all Append() calls
I20260927 18:51:22.580420 20619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc: No bootstrap required, opened a new log
I20260927 18:51:22.580906 30333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:51:22.582044 20518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a72c292436404388be4ea4803d19f096"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36540
I20260927 18:51:22.582289 20695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925: No bootstrap required, opened a new log
I20260927 18:51:22.582252 20518 raft_consensus.cc:482] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:51:22.582401 20518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:22.583778 20518 raft_consensus.cc:504] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } }
I20260927 18:51:22.584663 20518 leader_election.cc:288] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [CANDIDATE]: Term 1 election: Requested vote from peers db915133f79a4770a095673dd1f7bbdc (127.29.159.125:44131), 24c35297ea70497da8638174d3955925 (127.29.159.124:33909)
I20260927 18:51:22.582237 20476 master_runner.cc:186] Error getting master registration for 127.29.159.125:44131: OK, instance_id { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" instance_seqno: 1790535082298087 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:51:22.585912 20476 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:51:22.586952 20695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } }
I20260927 18:51:22.587200 20695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c35297ea70497da8638174d3955925, State: Initialized, Role: FOLLOWER
I20260927 18:51:22.587555 20695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } }
I20260927 18:51:22.588047 20722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } } }
I20260927 18:51:22.588193 20722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:22.588338 20619 raft_consensus.cc:348] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } }
I20260927 18:51:22.588553 20619 raft_consensus.cc:729] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db915133f79a4770a095673dd1f7bbdc, State: Initialized, Role: FOLLOWER
I20260927 18:51:22.588851 20619 consensus_queue.cc:260] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } }
I20260927 18:51:22.588963 20695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:22.589041 20668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a72c292436404388be4ea4803d19f096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "24c35297ea70497da8638174d3955925"
I20260927 18:51:22.589248 20668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:22.589382 20723 sys_catalog.cc:455] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } } }
I20260927 18:51:22.589583 20723 sys_catalog.cc:458] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:22.591163 20668 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72c292436404388be4ea4803d19f096 in term 1.
I20260927 18:51:22.591763 20493 leader_election.cc:302] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c35297ea70497da8638174d3955925, a72c292436404388be4ea4803d19f096; no voters:
I20260927 18:51:22.592120 20703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:51:22.592350 20703 raft_consensus.cc:686] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [term 1 LEADER]: Becoming Leader. State: Replica: a72c292436404388be4ea4803d19f096, State: Running, Role: LEADER
I20260927 18:51:22.592736 20703 consensus_queue.cc:237] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } }
I20260927 18:51:22.594513 20703 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a72c292436404388be4ea4803d19f096. Latest consensus state: current_term: 1 leader_uuid: "a72c292436404388be4ea4803d19f096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } } }
I20260927 18:51:22.594621 20703 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [sys.catalog]: This master's current role is: LEADER
W20260927 18:51:22.595153 20735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:51:22.595234 20735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 18:51:22.598963 20715 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:51:22.599035 20715 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:51:22.590591 20619 sys_catalog.cc:564] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:51:22.590636 20593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a72c292436404388be4ea4803d19f096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "db915133f79a4770a095673dd1f7bbdc"
I20260927 18:51:22.599545 20593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:51:22.600943 20593 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72c292436404388be4ea4803d19f096 in term 1.
I20260927 18:51:22.598119 20626 master_runner.cc:186] Error getting master registration for 127.29.159.125:44131: OK, instance_id { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" instance_seqno: 1790535082298087 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:51:22.601647 20626 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:51:22.601756 20739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:51:22.602777 20739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:51:22.603878 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.586790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.29.159.126:46723)
0927 18:51:22.586916 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.586921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.591975 (+ 5054us) server_negotiation.cc:407] Connection header received
0927 18:51:22.592120 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.592124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.592181 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.592270 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.593175 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.593895 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.595049 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.595345 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.595415 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.597382 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.597386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.597389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.597416 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.599170 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.602252 (+ 3082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.602256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.602258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.602355 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.602677 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.602681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.602683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.602901 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.603072 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.603626 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.603738 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:51:22.612984 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.594829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54678 (local address 127.29.159.124:33909)
0927 18:51:22.594937 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.594940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.604592 (+ 9652us) server_negotiation.cc:407] Connection header received
0927 18:51:22.604769 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.604772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.604826 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.604878 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.605771 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.606399 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.607249 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.607576 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.607693 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.609112 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.609117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.609120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.609149 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.610988 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.611633 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.611637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.611639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.611720 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.612032 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.612035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.612037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.612244 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.612342 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.612687 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.612809 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:51:22.617043 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:22.591154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.29.159.125:44131)
0927 18:51:22.591317 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.591321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.601536 (+ 10215us) server_negotiation.cc:407] Connection header received
0927 18:51:22.601583 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.601585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.601642 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.601723 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.602945 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.603814 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.604999 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:22.605305 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:22.605382 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:22.608839 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:51:22.608843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:51:22.608845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:51:22.608874 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:51:22.610988 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.615582 (+ 4594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.615587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.615590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.615703 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:51:22.616052 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:51:22.616057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:51:22.616060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:51:22.616286 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:51:22.616399 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:22.616743 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 18:51:22.616868 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 18:51:22.618709 20753 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:51:22.621038 20753 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:51:22.622310 20593 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [term 1 FOLLOWER]: Refusing update from remote peer a72c292436404388be4ea4803d19f096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:51:22.622345 20668 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [term 1 FOLLOWER]: Refusing update from remote peer a72c292436404388be4ea4803d19f096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:51:22.623049 20703 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:51:22.623510 20703 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:51:22.626858 20723 sys_catalog.cc:455] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a72c292436404388be4ea4803d19f096. Latest consensus state: current_term: 1 leader_uuid: "a72c292436404388be4ea4803d19f096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } } }
I20260927 18:51:22.626992 20723 sys_catalog.cc:458] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:22.627297 20722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a72c292436404388be4ea4803d19f096. Latest consensus state: current_term: 1 leader_uuid: "a72c292436404388be4ea4803d19f096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } } }
I20260927 18:51:22.627410 20722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:22.637878 20736 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a72c292436404388be4ea4803d19f096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } } }
I20260927 18:51:22.638043 20736 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:22.638166 20752 mvcc.cc:205] Tried to move back new op lower bound from 7334031698413223936 to 7334031698303377408. Current Snapshot: MvccSnapshot[applied={T|T < 7334031698413223936}]
I20260927 18:51:22.639212 20722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a72c292436404388be4ea4803d19f096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } } }
I20260927 18:51:22.639333 20722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:22.639469 20755 mvcc.cc:205] Tried to move back new op lower bound from 7334031698413223936 to 7334031698303377408. Current Snapshot: MvccSnapshot[applied={T|T < 7334031698413223936}]
I20260927 18:51:22.640626 20703 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a72c292436404388be4ea4803d19f096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } } }
I20260927 18:51:22.640739 20703 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096 [sys.catalog]: This master's current role is: LEADER
I20260927 18:51:22.643386 20723 sys_catalog.cc:455] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a72c292436404388be4ea4803d19f096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72c292436404388be4ea4803d19f096" member_type: VOTER last_known_addr { host: "127.29.159.126" port: 46723 } } peers { permanent_uuid: "db915133f79a4770a095673dd1f7bbdc" member_type: VOTER last_known_addr { host: "127.29.159.125" port: 44131 } } peers { permanent_uuid: "24c35297ea70497da8638174d3955925" member_type: VOTER last_known_addr { host: "127.29.159.124" port: 33909 } } }
I20260927 18:51:22.643494 20756 mvcc.cc:205] Tried to move back new op lower bound from 7334031698413223936 to 7334031698303377408. Current Snapshot: MvccSnapshot[applied={T|T < 7334031698413223936}]
I20260927 18:51:22.643540 20723 sys_catalog.cc:458] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:51:22.647117 20739 catalog_manager.cc:1347] Generated new cluster ID: 0021206facbe4c38bee940191aefd22a
I20260927 18:51:22.647190 20739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:51:22.671381 20739 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:51:22.672941 20739 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:51:22.691190 20739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a72c292436404388be4ea4803d19f096: Generated new TSK 0
I20260927 18:51:22.694450 20739 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:51:22.738732 20547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:22.736377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.29.159.126:46723)
0927 18:51:22.736499 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.736503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.737609 (+ 1106us) server_negotiation.cc:407] Connection header received
0927 18:51:22.737812 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.737815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.737868 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.737923 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.738596 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36564: BlockingRecv error: recv got EOF from 127.0.0.1:36564 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 18:51:22.740135 20625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:22.736007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.29.159.125:44131)
0927 18:51:22.736161 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.736166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.739219 (+ 3053us) server_negotiation.cc:407] Connection header received
0927 18:51:22.739384 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.739387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.739475 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.739527 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.740017 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46704: BlockingRecv error: recv got EOF from 127.0.0.1:46704 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 18:51:22.741516 20698 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:22.737091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.29.159.124:33909)
0927 18:51:22.737200 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.737206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.740623 (+ 3417us) server_negotiation.cc:407] Connection header received
0927 18:51:22.740787 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.740790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.740841 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.740891 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.741388 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54682: BlockingRecv error: recv got EOF from 127.0.0.1:54682 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 18:51:22.745906 20625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:22.744373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.29.159.125:44131)
0927 18:51:22.744493 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.744497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.744975 (+ 478us) server_negotiation.cc:407] Connection header received
0927 18:51:22.745182 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.745185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.745258 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.745310 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.745807 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46712: BlockingRecv error: recv got EOF from 127.0.0.1:46712 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 18:51:22.750605 20625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:22.749251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.29.159.125:44131)
0927 18:51:22.749345 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.749349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.749798 (+ 449us) server_negotiation.cc:407] Connection header received
0927 18:51:22.749961 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.749964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.750021 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.750068 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.750514 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46722: BlockingRecv error: recv got EOF from 127.0.0.1:46722 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 18:51:22.756227 20625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:22.754703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.29.159.125:44131)
0927 18:51:22.754800 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.754804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.755273 (+ 469us) server_negotiation.cc:407] Connection header received
0927 18:51:22.755542 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.755545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.755596 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.755647 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.756130 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46732: BlockingRecv error: recv got EOF from 127.0.0.1:46732 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 18:51:22.764894 20625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:22.763175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.29.159.125:44131)
0927 18:51:22.763358 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.763364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.763943 (+ 579us) server_negotiation.cc:407] Connection header received
0927 18:51:22.764133 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.764137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.764188 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.764238 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.764782 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46742: BlockingRecv error: recv got EOF from 127.0.0.1:46742 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260927 18:51:22.777280 20625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:22.775538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.29.159.125:44131)
0927 18:51:22.775666 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.775670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.776246 (+ 576us) server_negotiation.cc:407] Connection header received
0927 18:51:22.776481 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.776485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.776541 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.776595 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.777166 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46758: BlockingRecv error: recv got EOF from 127.0.0.1:46758 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 18:51:22.797946 20625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:22.796102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.29.159.125:44131)
0927 18:51:22.796234 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.796239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.796916 (+ 677us) server_negotiation.cc:407] Connection header received
0927 18:51:22.797150 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.797154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.797209 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.797264 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.797826 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46770: BlockingRecv error: recv got EOF from 127.0.0.1:46770 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 18:51:22.834512 20625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:22.833094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.29.159.125:44131)
0927 18:51:22.833283 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.833288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.833445 (+ 157us) server_negotiation.cc:407] Connection header received
0927 18:51:22.833666 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.833671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.833736 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.833826 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.834365 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46786: BlockingRecv error: recv got EOF from 127.0.0.1:46786 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 18:51:22.902599 20625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:22.901251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.29.159.125:44131)
0927 18:51:22.901392 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:22.901396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:22.901670 (+ 274us) server_negotiation.cc:407] Connection header received
0927 18:51:22.901813 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:22.901816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:22.901866 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:22.901957 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:22.902491 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46798: BlockingRecv error: recv got EOF from 127.0.0.1:46798 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 18:51:23.034185 20625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:23.032882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.29.159.125:44131)
0927 18:51:23.033047 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:23.033054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:23.033254 (+ 200us) server_negotiation.cc:407] Connection header received
0927 18:51:23.033431 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:23.033434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:23.033485 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:23.033582 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:23.034044 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46804: BlockingRecv error: recv got EOF from 127.0.0.1:46804 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 18:51:23.293730 20625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:51:23.292509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.29.159.125:44131)
0927 18:51:23.292656 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:23.292661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:23.292784 (+ 123us) server_negotiation.cc:407] Connection header received
0927 18:51:23.292957 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:23.292961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:23.293044 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:23.293160 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:51:23.293613 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46810: BlockingRecv error: recv got EOF from 127.0.0.1:46810 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:51:23.596540 20735 catalog_manager.cc:1259] Loaded cluster ID: 0021206facbe4c38bee940191aefd22a
I20260927 18:51:23.596625 20735 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925: loading cluster ID for follower catalog manager: success
I20260927 18:51:23.598294 20735 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925: acquiring CA information for follower catalog manager: success
I20260927 18:51:23.599623 20735 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24c35297ea70497da8638174d3955925: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:51:23.622493 20753 catalog_manager.cc:1259] Loaded cluster ID: 0021206facbe4c38bee940191aefd22a
I20260927 18:51:23.622618 20753 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc: loading cluster ID for follower catalog manager: success
I20260927 18:51:23.624445 20753 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc: acquiring CA information for follower catalog manager: success
I20260927 18:51:23.625658 20753 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db915133f79a4770a095673dd1f7bbdc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:51:23.811538 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:23.807811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.29.159.125:44131)
0927 18:51:23.808060 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:23.808064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:23.808147 (+ 83us) server_negotiation.cc:407] Connection header received
0927 18:51:23.808270 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:23.808272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:23.808399 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:23.808523 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:51:23.808736 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:23.809252 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:23.810122 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:23.810338 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:23.810416 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:23.810666 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:51:23.811092 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:51:23.811184 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:23.811236 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 18:51:23.811282 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
I20260927 18:51:23.818696 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:23.814527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.29.159.124:33909)
0927 18:51:23.814800 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:23.814805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:23.814825 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:51:23.814986 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:23.815009 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:23.815182 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:23.815317 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:51:23.815706 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:23.816279 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:23.817171 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:23.817403 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:23.817481 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:23.817761 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:51:23.818252 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:51:23.818358 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:23.818424 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 18:51:23.818491 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":66,"threads_started":1}
I20260927 18:51:23.826462 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:23.821842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.29.159.125:44131)
0927 18:51:23.822001 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:23.822008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:23.822242 (+ 234us) server_negotiation.cc:407] Connection header received
0927 18:51:23.822443 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:23.822446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:23.822512 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:23.822593 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:51:23.822874 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:23.823650 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:23.825053 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:23.825399 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:23.825516 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:23.825975 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:51:23.826046 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:51:23.826115 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:23.826264 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 18:51:23.826324 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:51:23.827052 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:23.822242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.29.159.124:33909)
0927 18:51:23.822346 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:23.822350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:23.822626 (+ 276us) server_negotiation.cc:407] Connection header received
0927 18:51:23.822806 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:23.822813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:23.822869 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:23.822913 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:51:23.823253 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:23.824021 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:23.825102 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:23.825424 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:23.825516 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:23.825960 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:51:23.826046 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:51:23.826117 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:23.826875 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 18:51:23.826930 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 18:51:23.827271 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:51:23.821806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.29.159.126:46723)
0927 18:51:23.822046 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:51:23.822051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:51:23.822130 (+ 79us) server_negotiation.cc:407] Connection header received
0927 18:51:23.822471 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:51:23.822475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:51:23.822650 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:51:23.822762 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:51:23.822988 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:23.823819 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:23.825586 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:51:23.825809 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:51:23.825921 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:51:23.826257 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:51:23.826638 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:51:23.826743 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:51:23.827029 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 18:51:23.827094 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":58,"threads_started":1}
I20260927 18:51:23.830438 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 20476
I20260927 18:51:23.836879 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 20551
I20260927 18:51:23.842718 30333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskclkeqe/build/debug/bin/kudu with pid 20626
2026-09-27T18:51:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3140 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6272 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (282693 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST